Newsletter – Why Architecture

This is the first newsletter article proper. I will try to keep this brief, but If there is sufficient interest, I could spend some more time/words on the topic. Our topic today, is an important question, and one that architects will face from time to time. Here is a recent article that I wrote on this subject. Do read it if you can, as it covers details that I will not rehash here.

But in answer to the question, my response has always been that the drivers for IT architecture are, mindset and environmental imperatives. The first is proactive, while the second is reactive. As you can expect, I recommend the former (mindset) over the latter, because it instils architecture from the get-go. Such organisations seek excellence, are in for the long haul, or seek to rapidly gain market-share in an existing market. Because of their ambitions, such businesses will prioritise strategy, even in MVPs.

Almost always, this strategic view percolates from champions in the highest levels of the business (c-suites typically), to the IT area. Because the influence is coming from the highest levels, the effects are pervasive and resilient, resulting in an IT culture that is closely aligned with the business aspirations. In this kind of enterprise, what one may label as MVP, is actually an optimal release for the time, constraints and environment of the business. Because it is under-girded by sound business and architectural thinking. This is not true for all organisations, particularly those that employ architecture reactively.

Some other organisations introduce architecture, as a response/reaction to changes in their environment. The stimuli that I identify are, scale, complexity, and risk. Some experts also mention cost, but I think cost is a direct consequence of the three primary imperatives. One is unlikely to see costs going south while any of these are going north. When things get more complex, scope multiplies, or risks (regulatory, financial, legal, others) emerge, most organisations will wake up to the need to engage staff outside of day-to-day operations. Unfortunately, this is often problematic, because governance becomes a big issue.

Folks who previously made their own decisions, now have to discuss it, and perhaps, submit it for oversight by architects. This can create tension, resistance and in some cases conflict, with vassals overseeing strongholds of opposition; overt or covert. Secondly, because architecture has not evolved as a culture, when the immediate need has been addressed, it is easy for those organisations to regress on the discipline of architecture. In the short term, it saves on the stress and tension that the introduction of architecture creates. In the long term, of course, the problems will re-emerge, having evolved into more complex and entrenched threats.

One may ask: what to do, if the architecture function is not yet present in our organisation? My advice is to identify a sponsor that you can communicate with directly. Preferably someone higher up (quite😀) in the organisational hierarchy, and preferably on the business side. Get to know the vision, targets and roadmap of the organisation; from a business perspective. Do your diligence to identify any of the motivations that I mentioned above for mindset-driven IT architecture. Now explain, in simple language, how taking a strategic view that aligns technology change with business change, at every step of the roadmap, will be salutary to success. Continue to state how this can be facilitated by the introduction of architecture.

As one who has led a startup, I can tell you first-hand that most business leaders see tech as an enabler only. Therefore, you must arm yourself with numbers and other quantitative details. Imagine that my goal as CEO is to make 170x in value within a given time, and I have investors expecting 38x of that back in the same time. Your suggestions must show how we can either overshoot the target now, or miss it “mildly”, but secure significant and increasing gains in the near future. If I don’t get that empathy, I am unlikely to consider what you are offering. That is the red, proactive pill.

If that fails, because no one gets the picture, try the blue (reactive) pill as a last resort. Identify existing or potential change imperatives, as I mentioned earlier, and use that to argue for advance preparation. Such pre-emptive action requires a stepping above the business-as-usual (BAU), looking to the horizon and putting plans in place to harness everything in the enterprise, in synergy, to manage anticipated changes. Once again, you will need to demonstrate how the introduction of a strategic (architecture) rather than tactical approach, creates value for the organisation. If you succeed with either pill, be prepared to lead the change. Trust me, it will be challenging, but extremely rewarding, in the short and long terms.

Before I go any further, I need to point out that the focus is not on technical or application architecture, which typically approach solutions from a technology perspective. Rather, it will be on solution and integration architecture. The two roles are very similar, and both work closely with the business, various architecture teams, and the core technical/delivery teams. Henceforth, I will use solution architect or solution architecture as generic terms, for both. I found this video by Marco Lenzo, most enlightening, in illustrating this bridging function provided by solution architects. Marco has several short videos on key technologies, which I find to be handy time-savers.

Solution architecture (SA) has some overlap with enterprise architecture (EA). However, as organisations scale up, the distinction of function is cleaner and the overlap decreases. In my opinion EA, sets the stage for architectural involvement within an organisation (enterprise), whereas SA works with the business and technical teams to deliver changes, within the constraints that EA has previously agreed, and continually evolves, with the business. As you can see from the diagram above, SA has a vital role in keeping the technical/delivery functions aligned with the business.

I initially planned for 500-word articles, but having reviewed the number of topics, and the possibility that more may be introduced, I am doubling that number, to keep the lifetime of the newsletter manageable. Please let me have your comments and feedback for this week. I will respond to any comments, and God willing (GW), continue the discourse in two weeks’ time.

Adios for now. 🙏🏿👍🏿

Thinking of AI, LLMs and Gen-AI

Like most folks on the planet right now, AIs have been one of my major areas of interest and attention; at least in the last two years. Yes, I caught on late. But it was not for want of trying. It was just that the first wave of AIs, were too low-level for me. The theoretical aspects of the courses run by Google and the University of Helsinki were accessible to me, but the maths blew me away completely. Now I wished I had paid more attention in maths class 🙂

Well, all is not lost, and as often happens in IT, the second wave, or should I say evolutionary kind of the technology abstracts away much of the complexity of the earlier kind. Yes, you lose access to a lot of the fine-grained controls, but hey, how many folks care about the difference between Transformers and the CNNs and RNN, or what about the algorithms used by AMAs for prompt reliability? Or this simple equation from LLM calibration q^=Softmax(Wp^+b)?

Have a good look at the headline image. Those three amigos are your friends in the world of AIs and human interfaces to that complex nether world. I program in quite a few languages, including Python, on the side, but I never quite paid attention to it until recently. If you want to work with AIs, Python is your easy way in. The ecosystem is extensive and with several layers of abstraction that make it easy for almost anyone to enter at a level that is comfortable and accessible.

The other two are Swiss-army knives in their own right. I will not waste time describing them, as you will find a plethora of material on them on the Internet. Suffice to say that LlamaIndex allows you to easily collect data (documents, APIs, SQL, etc.) and build an index. It relies on LangChain for the indexing. LangChain is like a workflow engine and connector that provides easy interfaces (facades) to the LLMs and other tools. Both are available as libraries in Python.

A big shout-out to Wenqi Glantz for an excellent intro that I was able to implement within a few minutes. I intend to modularise your solution by separating the indexing function and making it an event-driven daemon, since the Gradio interface only needs to know where the index is and not create it each time. Gradio is another Python library that makes it easy to create a web UI, in this case, for building a chatbot. Thanks also for Confluence example, it should be adaptable for JIRA as well; both of those have rather poor search/introspection facilities.

Honestly, we are at the point now where most folks, in the tech sector at least, should be able to understand and use AIs. The Python programming language is a great place to start, and though I have mentioned only three libraries today, there are several others out there, each providing simple interfaces to advanced capabilities. Now is a great time to get into AIs. The technology has moved on from white-coats and geeks. You still get to see and understand some of the fundamentals, before it is all hidden away in later generation services and solutions. Take the plunge; I am sure you will enjoy the journey, and there is so much to learn and explore!

ChatGPT Prompting for Architects and Executives

Introduction

In chapter six of Lewis Carroll’s “Alice’s Adventures in Wonderland,” under a section, titled “Pig and Pepper.”, the following dialogue plays out:

Alice: Would you tell me, please, which way I ought to go from here?
The Cheshire Cat: That depends a good deal on where you want to get to.
Alice: I don’t much care where.
The Cheshire Cat: Then it doesn’t matter which way you go.

There is a lesson here for interacting with emerging machine interfaces of Large Language Models (LLM) and Generative Artificial Intelligence (Gen-AI) like ChatGPT. The way we interact with computers has evolved dramatically in the last decade, to the point where one can begin to conceive of Alice’s conversation with the Cheshire cat. However, much as it was with the Cheshire cat, you still need to know where you want to go in order to benefit from directions.

Let’s imagine that you are looking for information about the topography of Berlin, Germany, and not the whole of Europe. Well, it helps to know that your target lies somewhere between Russia, the North Sea, the Mediterranean and Lebanon (context), since there is another Berlin in Ohio (US). It is useful to understand that the focus is geographical rather than political (role), that the message is in a form suited for delivery at the UN rather than the Republican conference (tone), and the format is as a speech rather than a manifesto (format). These foundations are effective for all ChatGPT interactions, whether you’re an IT architect navigating complex technical issues or a C-Suite executive seeking strategic insights.

There are many comprehensive blogs, articles, books and other materials you can read about writing effective prompts. I find that many are quite technical, and they cover details and scenarios that most folks will never need, in work, or everyday life. What I have done is to simplify things and give you the bare essentials. A simple formula that gives you the maximum revenue-to-investment ratio. The acronym CRAFT spells out: Context, Role, Action/question, Format and Tone. These are the fundamental components of every ChatGPT prompt. By applying these to your prompts, you will get much better results, and useful feedback for improving future prompts.

Context

Context is the backstory to your dialogue; it gives depth to your prompt. It provides the AI with the necessary background information to better understand your query fully. For example, if your question is about technology threats and trends, the prompt could start like this: “In the rapidly evolving tech industry, where startups are disrupting traditional players…”

Role

Here is the best way to think of the role. Think; in real life, who would I prefer to answer this question: a lawyer, plumber, dietician, legislator, soldier? The role instructs the machine on the overall nature and perspective of the response, including any inherent biases. For instance, if you’re a COO architect seeking technical insight, your prompt, continuing from the Context example above, might be of the form: “you are an experienced Cloud infrastructure expert, with several certifications and hands-on experience of Azure …”.

Action/Question

Instructions in a ChatGPT prompt are the most important component. They need to be clear, concise, and unambiguous. Brevity might be an asset here, the clearer your instructions, the more accurate and relevant the AI’s response. A compatible question to the example given in the Context could be: “what are the emerging cybersecurity threats for Azure IaaS in 2023?”

Format

This determines the presentation of the output/response of ChatGPT. It could be a detailed analysis, a concise summary, a table, bullet points, a step-by-step guide, or a combination of things? Use simple language to tell the AI how to deliver the information in a way that suits your purposes. Once again, using our example from the Context, the format could be: “a short summary with bullet points of the 5 most significant concerns”.

Tone

The tone of your prompt sets the mood for the conversation. If you were to advise the role how best to engage with the target audience; would it be technical, formal, humorous, conversational, expert, witness, informative, urgent, casual, professional, legalistic, etc. It could also be a combination, for example professional+humorous. Extending our Context example, the tone could be: “speaking with an informative tone”, since the COO does not intend to mask their unfamiliarity with the subject.

Extra Tips

  • Use simple and precise language; imagine that you are conversing with a 10-year-old Einstein.
  • Evolve the prompt, changing the keywords you use for tone and format, in order to get an output that suits your purpose.
  • Always leave the action/question till last, as it can help reduce verbosity and improve accuracy of the output.

Conclusion and Credits

In the ever-evolving landscape of AI, mastering the art of ChatGPT prompting is a skill that can elevate your interactions with technology. IT architects can it to seek solutions to complex technical challenges, while C-Suite executives can extract strategic insights for decision-making. From general technical information to specific market analysis and research, the applications are limitless. I am convinced that we are entering a new age of man-machine relationship, and those that learn and adapt will reap significant benefits in the future.


PS: I used an approach called “Generative Knowledge” to elicit an outline and the initial draft for this article. I had to refine the prompt a few times to get things right. The final LLM output benefited from some massaging, but all-in, it saved me a lot of time, and I hope that I managed to impart some useful information to you.

Advanced PayPal Checkout Simplified

If you have an eCommerce site that is growing, you may have noticed that some recurring expenditure is also growing. One such is the cost of transactions, in this case, commissions or transaction charges on sales. The costs are very similar, whether you are using PayPal, Stripe, Flutterwave or some other payment gateway.

The bad news is that you cannot (easily) cut out payment gateways, they are a bit like taxes. 🙁 The good news is that you can reduce the amount you shell out per transaction. One option is to integrate with the payment gateway directly, rather than using a third party product.

Third party products are great, especially the free ones. However, as we all know, there is no free lunch. While they get you off the ground quickly, the convenience has a cost attached: higher per-transaction costs. After a while, and as your sales volumes increase, those pennies add up. Furthermore, these products often impose significant constraints on how the products and the shopping cart are displayed on screen.

Sooner or later, you get to the point where you may want to go ‘pro’ for greater artistic freedom, to gain access to premium features, or to reduce the transaction charges. But ‘pro’ versions can be quite pricey for small-holders or those running a side-hustle. Almost all are subscription based, and the fees are paid annually, in advance. In essence, you pay upfront, whether you make money or not.

I had the same challenge, and decided to have a look at what it would take to integrate directly with PayPal. For me, the motivations were costs and design freedom. It took a while, but that was because I started off reading the PayPal docs, which, in my opinion, could have been better written. However, I soon found alternative resources on the web that made the process clear and simple.

Using those sources, I have created a repository with a simple, customisable shopping cart that integrates with PayPal. Why create a shopping cart, aren’t there tons of those already out there? Well, all the ones I saw were integrated with eCommerce platforms such as WooCommerce, EDD, etc., which meant that you need to install at least one other plugin to get the cart working. I wanted something generic, independent and simple, with all the skeleton visible and tweakable. It is now ready, and available to the public, gratis, on GitHub.

With a few minor edits (URLs and PayPal client Id), you can get this solution working in minutes. I have used plain and simple JavaScript, HTML and CSS, making it very easy to customise the functionality or redesign the visuals. Click on the GitHub link below to view the repo, read the description, take a copy and see how quickly and easily you can be up and running. If you would like a WordPress plugin to make things even easier, give me a ping.

https://github.com/LanreOyewole/PayPalIntegration/tree/main

The Leadership Quadrant

English is one of the world’s most widely spoken languages.  The language has evolved significantly over time, transforming from a cultural to a functional language.  It is now a global communication tool, rather than a codification and expression of the values and practices of the English people.  I am a first-generation immigrant, originally from Nigeria and of the Yoruba tribe.  However, some people might consider me to be English, simply because I speak the language and I presently reside in England.

One can be English without any reference to culture.  Not so, my mother tongue, Yoruba, which is still very much cultural.  The English colonialists referred to the Yoruba as a tribe or ethnic group.  Actually, Yoruba is an identity; it spans geographies and crosses racial boundaries.  Unlike English, you cannot be ‘Yoruba’ without a sound understanding of the language and its many nuances, inferences and practices.  The culture and the language are inseparable.  One does not have to be black to be Yoruba, and one could be of black Yoruba parents and not be Yoruba.

It is therefore not surprising that, when translating between Yoruba and English, most words do not have an equivalent.  But there are a few notable exceptions.  One such is the word “leader” in English.  The equivalent in Yoruba is the word “asiwaju”, often shortened, as is common in Yoruba, to the form “asaaju”.  It is very interesting to note that in Yoruba and in English, the word is both a noun and a verb.  In both languages, it is mostly used in the active sense, i.e., as a verb.

In his world-famous book, “The Road Less Travelled”, M. Scott Peck tells us: “Love is as love does. Love is an act of will — namely, both an intention and an action”.  One could say the same for the leader and leadership.  Leadership is not an emotion.  It is not goodwill or good intentions.  Leadership is what a leader does for the good and progress of their community.

Real leaders are an asset to a community.  However, in addition to any benefits that are imparted, the connection to, and immersion within, the community, are important.  The leader must be a part of the community, else they become a benefactor, a sponsor, or a controller; in order of increasing detachment.  He or she feels the community’s emotions and their aspirations.  They understand the problems and difficulties inherent in the status quo, and they know what good might look like for the community.

It is important to stress the subjectivity of “good”.  What one community considers to be progress might be a step down from the status quo for another.  That is why it is important that the leader has unfiltered access to the community’s concept of good.  They also need to be aware of the gap between the community’s current and desired future states.  Paradoxically, the aspiration to close this gap also presents a quandary. It is allegorical of a critical crossroads where every choice, including hesitation, holds significance.  Without change, the undesirable status quo perpetuates, but either delay or a wrong step could plunge the community to even greater depths of despair.  Interestingly, this inertia presents a potential leader with a prime opportunity.

Sometime in 2022, I had the good fortune to be invited by my friend @Igbaun Okaisabor, to a talk given by @Soji Apampa on the topic of Professional Ethics and Morals.  The session was part of a series of talks organised by the Kaiser Foundation for Social Development (KFSD).  One major takeaway for me was the fact that every whole is a product of the synergies between the parts.  Good, proactive parts are required to craft a sound and progressive whole, whether that be a country, organisation, institution or neighbourhood.

Soji’s message was that, at all times and in all places, we are all potential leaders, having the choice to be active, indifferent or detached.  It was enlightening.  I gained new perspectives on latent tensions between each one of us and our environments. It was also challenging.  It reminded me that indifference or passivity is not a true option for any, except the hypocrite.  There is no neutral ground.  We choose, even in apathy, to be part of the problem and to part ways with the agents of positive change.

He deployed illustrations to explore leadership, two of which struck deep resonance with me.  The first was the boiling frog apologue, and the second was a quadrant on ethical leadership.  The apologue is reactive or reflexive.  It speaks to how we respond when thrown in with bad apples.  The quadrant is proactive.  It shows how we all have some measure of authority or influence, and we can impact our environment in some way.  In the illustration below, I have adapted the quadrant, merging ethics and morality on one axis and setting them against strength and resilience of the person on the other axis.

This adapted quadrant reveals four forms in which individuals interact with their environment (context).  They are; as a figurehead, a manager, a hypocrite or an inspirator.  These are types of roles that a person can play in one context at a given time.  However, it is important to state that, at that time, the person will be playing different roles in the various contexts in their life.  For example, a person might be a manager at home, a figurehead in church, a hypocrite in the political party, but an inspirational leader in the community football team.  The role does not define the person, it merely describes how they engage within a certain context, and at a given time.   

Soji stresses the importance of a proactive exertion of influence.  But he also states that ethics and morality must form the core of “good” personal or corporate activity.  According to him, actions may seem right, but without moral undergirding, they lack lasting goodness.  Consider the case of a truck driver aiding stranded bus passengers; initially commendable, until it is revealed that each passenger was obliged to make prior payments in cash or property.  True goodness has its roots in ethics and morality.

For the perceptive, Soji’s main thrust was to our collective midriff.  If the beam was on black Africa, the spotlight was on Nigeria. Without mentioning Succubus by name, he challenges the shallowness or absence of leadership in Nigeria.  Nigerians often speak of a shadowy unelected cabal that, in the popular imagination, runs a parallel government to the elected executive. This is a crystallisation of the portentous “Unknown Soldier” album, released by the late Nigerian musical icon Fela Anikulapo Kuti.  The mononymous Fela spoke of the failure of critical parts of the governance infrastructure, culminating in what he termed ‘unknown government’.  The ship of state being on the high seas, but there is no one on the bridge, only smoke and shadows.

One might glibly say that this is to be expected where figureheads predominate in the leadership of a given context.  But from Soji’s thesis, we gain an alternative perspective:  that this is an indication of a high density of ‘parochial cultures’ throughout the context.  Because, as he asserts; we all are potential leaders.  Each one of us has a scope of influence, by default.  We choose, either to impact our environment, or like the frog try to be conformed to it: good or bad.  In the illustration, the figurehead sits at the bottom and has the least influence of all the types, as described by the shape and area it occupies.  This is the default setting for most of us, until we choose to engage and take responsibility for change.

The figurehead is an exemplar of apparitional power and typifies a deviance that Robert K. Merton identifies in his Strain theory.  This is a contradiction, because the person behaves as if they have no influence.  While playing this role, we interact with others as one that has given up on the goals and values of the context.  The figurehead exhibits a lack of conviction or strength of character, in the context.  In an organisation or other formal structure, such a person may have been around for quite a while, but it is not apparent from their interactions.  They may be a beneficiary of undeserved elevation, or have had expectation or responsibility foisted upon them, but fail, or refuse to take up responsibility.

As a consequence, the figurehead comes across as aloof or unaware, and is often viewed as distant or lazy.  They simply show up.  Like a billboard, they point at something, but are totally disconnected from the reference. They appear unconcerned about honesty and integrity, personally or in the collective. This role is a magnet for those at the very bottom of the personality and conviction axes.  These persons have also given up on the values of the context and simply survive for as long as they can.  The figurehead leaves them alone, lacking the desire or will to challenge or inspire others, and so inertia perpetuates in the context.

However, this lack of engagement means that the figurehead is caught in a hell of sorts.  Without change, dissatisfaction will continue and frustrations will grow, further undermining the stability of the context.  But, we all start somewhere, and there is good news ahead.  Most contexts have few long-standing figureheads.  People start out as figureheads in a new context, but they soon move up.  Over time they grow in confidence and influence, evolving to exert their influence in a number of ways.

Some are born great, some achieve greatness, and some have greatness thrust upon them
– William Shakespeare: Twelfth Night

One evolutionary path for the figurehead is towards the hypocrite.  The hypocrite exerts powerful influence, but in a limited area.  He/she has a strong character (personality) but either lacks moral depth, or has limited overlap with the values that underpin the context.  It is not uncommon for hypocrites to be competent and even hardworking, but their disconnection and/or moral shallowness creates tension.  This person, by dint of their role or socialisation, knows what is right and good, but, either as an act of rebellion or for personal gain or comfort, chooses to disregard them.  If we operate as a hypocrite in a context, we succumb to domination by the ego.  Within that context, we would live, mostly to serve ‘me’, and it is often the case that we see the end as justification for the means. 

Love is the will to extend one’s self for the purpose of nurturing one’s own or another’s spiritual growth… We do not have to love. We choose to love.
– M. Scott Peck

Another evolutionary path for the figurehead is to become a manager.  Many more of us evolve to be managers rather than hypocrites, probably because it takes more energy and effort to be and sustain the hypocrite role. The manager is very connected with the context and shares its values.  She/he is one that has found resonance with, and is an exemplar of the values of a context.  Manager types will often be elevated by their context (community, organisation or group) to a position of relevance, prominence or responsibility.  When we function as a manager, we act as value custodians and gatekeepers of what has been achieved along the path to what the community considers to be good.

The manager is therefore a person who has similar aspirations of “good” as the community.  They hold themselves up to high standards within the community’s definition of good.  Unfortunately, this is not often supported with courage and resilience.  The manager type often constrains their ambition and scope of influence to like-minded people.  While they always mean well, their aspirations are tempered by a conservative pragmatism.  They tend to be risk averse and would rather tolerate the status quo than provoke unpredictable change.

Very few of us start out as an inspirator in any context.  Even for those positioned to lead from the outset, connections and relations must be established with those to be led.  Some time is required to develop the relationships that open up vistas for nurture.  It is important to mention that the opportunity for transformation into an inspirator is a transient portal.  When it opens, one must approach it with the utmost enthusiasm and an open mind.  Allow me to reframe the concept.  There are a variety of vehicles that will conduct a person to their appointment with destiny.  The hypocrite flies in on the wings of ego and an enlarged sense of self.  The manager arrives under the weight of the cares of the context and those in it.  Upon arrival, it is crucial that they both understand that the same vehicle may not be suitable for subsequent voyages.

There is a tide in the affairs of men, which, taken at the flood, leads on to fortune;
Omitted, all the voyage of their life is bound in shallows and in miseries.”
– William Shakespeare: Julius Caesar

There is a popular African proverb that has many attributions on the web.  It says, “if you want to go fast, go alone, if you want to go far, go together”.  If one obtains the opportunity to be an inspirator, that person must become comfortable with heterogeneous company.  It is not a solo trip, and you don’t get to choose your co-travellers.  Other changes are imperative. The manager must develop strength of character, determination and master how to engage with “strangers”.  The hypocrite must learn persuasion, soft power and the importance of compromise. 

Both must rise above their initial impetus and be prepared for further metamorphoses.  Ego, strength of character, energy, loyalty, value resonance and empathy will be needed to sustain the role of inspirational leader.  Those who manage the transformation get to leave enduring footprints in the sands of the context.  Others will lose the desire or means to remain, falling back into their previous role, and be soon forgotten.

The inspirator is strong on both axes of the quadrant.  They combine deep personal conviction of what is ethical and moral, great courage, imagination, empathy and passionate perseverance.  Inspirational leaders hold themselves to the highest standards of commitment, diligence and integrity, and challenge those around them to aspire to similar heights.  The core motivation of the inspirator is love.  When we choose to love in our context, we are elevated to the highest level of consciousness, as Steve Barrett says, in his  book, ‘A New Psychology of Human Well-Being’.  The book explores the concept of service consciousness, and can be understood as an exposition of Mahatma Ghandi’s saying: “the best way to find yourself is to lose yourself in the service of other”.

The Inspirator is a positive example to others.  They embody the values of the context and work continuously to evangelise and nurture those values throughout the context.  In almost every context, there will be those that just happen to be there, or show up, because options are few or non-existent.  People in this state are unreachable by the figurehead, hypocrite or manager.  These persons do not have the desire to reach out to others, and the values of the context lack the gravitational influence to draw them in.  Their lack of independent propulsion (drive, ambition, goals) means that change will have to be initiated and sustained by an external entity.  The hypocrite will not, because they add nothing to his/her corner.  The manager cannot, because they are averse to engaging with passivists or non-conformists.

The risks are high and the chances of success are slim.  Furthermore, significant resources will need to be invested over time, irrespective of outcome.  Only an inspirator can reach or help such person, despite the costs and risks, because they see beyond the present to the potential.  The inspirator is a kind of Jesus-bolt.  They act as a counterweight to the hypocrite, an ally and motivator to the manager, and a mentor to the figurehead.  Every context needs one or more inspirational leaders.  Without them, the lifespan of the context could be limited and/or the lifetime be marked by dissatisfaction and disquiet.

Love is patient, love is kind. It does not envy, it does not boast, it is not proud.
It is not rude, it is not self-seeking, it is not easily angered, it keeps no record of wrongs.
Love does not delight in evil but rejoices with the truth.
It always protects, always trusts, always hopes, always perseveres.
Love never fails. …

And now these three remain: faith, hope and love. But the greatest of these is love
I Corinthians 13

Fortunately, inspirators are not as uncommon as one might think.  When most people conceive of this type, their imagination conjures up the usual suspects: religious, political or military leaders, successful business people, daring entrepreneurs and adventurers, distinguished professionals or individuals singled out for public honour. In reality, most of us will play this role in one or more contexts during our lifetime.  If you are a loving parent, an unpaid caregiver, a dutiful child, a devoted spouse, a coach or counsellor, a teacher, a champion of causes less spoken for, or a selfless friend, you are an inspirational leader.  Yes, you are the one.  Surprised?  Well, I am not.  Here is a paradox. The inspirator, by very nature, does not see themselves as anything special, and might even question the plaudit.  Whereas some figureheads or hypocrites might be chafed by those who see them as otherwise of inspirational!

Before I conclude, I should add two qualifications. First; no human being is an inspirational leader in all contexts of their life at any point in time.  We all start out as figureheads in most contexts, we soon move on to become managers, which is why we endure in those contexts.  But in some places, we are hypocrites, and tellingly, those are contexts where, they don’t like us, we don’t like them, or both.  Second; only a few human beings live out their lives without ever being an inspiration in any context.  Almost everyone will combine the inspirator in one or more contexts with the other roles in all other contexts. 

We are all wired with the capacity to inspire others.  When and where we do, we are intentionally selfless, or as Scott Peck says; we choose to love.  While it is great that most contexts are dominated by the manager type, but that is like rowing the boat gently down the same river.  What if we could attach some skint-hulls to the sides, mount an engine on the aft and remodel the bow?  Perhaps we could have a Catamaran and head off into the bay, or venture out onto the open seas!  But for that to happen, we need inspirators on the bow and aft, as well as the port and starboard sides.  Ghandhi says, “be the change that you want to see”.  If we can inspire in one context, we could do same in two, three, or more.  It’s time to metamorphosise folks!  Let’s change the world, one context at a time.  Anyone for the aft?

HTTP Status Codes and Business Errors

In a three-tier architecture involving a frontend system (FE), a middleware layer (MW) and a backend system (BE), the FE sends requests via the MW to the BE.  However, the ML is not a pass-through component, rather it acts to decouple the FE and BE.  The MW also acts as a secondary security layer, inspecting and where appropriate rejecting improper or malicious requests.

This is a common arrangement, used wherever there is an ESB, legacy integrations and to varying degrees with microservices.  In normal operations (happy path scenarios), the arrangement works perfectly; the request is received from the FE, processed by the MW, and all being well, forwarded to the BE.  The arrangement also works well if/when technical errors are encountered.

It is useful to define what is meant by a technical error, to wit, any failure that occurs on account of the network, infrastructure, application code, or the basic structure of the message (headers, security or payload).  There are two scenarios that are pertinent to this discussion.  In the first, the FE sends an invalid request and the MW rejects it immediately, returning a 4xx error to the FE.

In the second, the FE sends a request, the MW accepts it and forwards same to the BE.  The BE rejects the request with either a 4xx or 5xx.  However, since the problem occurred in the hop between the MW and the BE, the MW remaps the error to a 5xx, which it returns to the FE.  The remapping aligns correctly with the semantics of HTTP status code groups 4xx and 5xx, given that the MW is not a pass-through.  See the diagram below for an illustration.

This may sound incorrect or misleading, however, in practical terms, any technical errors that occur after step 2 are most unlikely to have anything to do with the FE, given the functions performed by the MW.  A 4xx would imply that the FE was at fault, and as such, the error details would be required for it to rectify the problem, whereas in this case those details would only make sense to the FE.  There may be exceptions of course, but the approach is good for most cases.

But things start to creak a bit when business errors are encountered.  Once again, it is pertinent to pause and define what constitutes a business error.  In this context we refer to all errors that occur after the HTTP dialogue has completed successfully.  This implies that the FE has sent a request to the MW, the MW has examined the request and passed it to the BE.  The BE has in turn accepted the request without faulting any of the technical aspects: network, infrastructure, application code or message.

Having done some further processing of the message though, the BE identifies a problem with the request and reports this back to the MW.  The problem here is that the HTTP status codes did not have business errors in mind during compilation.  The status code groups (1xx, 2xx, 3xx, 4xx and 5xx) map very cleanly to genres of technical failures that could occur in the HTTP dialogue, at a technical level.

However, when it comes to business errors, there is no such clear mapping.  Indeed, the only status codes that could, with some imagination, be leveraged for business errors are: 400, 409 or 422.  The 400 code is a generic bad request; the 409 is an inconsistency of state, and the 422, introduced for WebDAV, indicates a semantic error.  Neither is perfect, but even with all these, there would still be a huge shortfall.  It is clear that there are far more than three kinds of business errors in the real world. 

To further complicate matters, there are those that will assert that business errors are not errors in the sense implied by the HTTP status codes and should use the 2xx group, because, technically, there was no error.  An alternative viewpoint emphasises the need to be aligned with the semantics of the HTTP status codes, and the fact that the MW is an active participant in the dialogue, as such, all errors, 4xx or 5xx ought to be mapped to a 5xx.

In a recent article, I spoke about ‘the fog of experience’, and how opinion and decisions become more nuanced as one travels down the paths and routes of our dynamic and ever evolving technological landscape.  This is one of those where the answer is not without contention and some subjectivity.  It is likely that various architects and organisations may differ on approach, but in each context, the opinions and final decision will be informed by knowledge and a pragmatic search and compromise on progress and value.

In my current context, the consensus is to accept the limitations of the HTTP Status codes and the fact that they are not about to change.  However, standards and consistency are important to quality and velocity in an Agile and product-centric service model.  For this reason, the decision was to recommend that all BEs capture business errors using the 422 HTTP status code.  All such errors are to be accompanied by a payload that describes the business error, and optionally, any details that could help the FE to rectify it.

The Fog of Experience

I still remember the early days when I first started out in tech. All answers were apparent and hardly was the ink dry on the request, and I was already delivering pieces of the solution. It was a kind of “Agile”; quick, and invariably, MVP. The success factors for me were essentially the functional aspects. Value was perceived from a tech perspective.

These days, so many answers are qualified, nuanced, and accompanied by caveats. Almost nothing is clear and certain. I have discovered that value is the prerogative of the customer, and he/she expresses it in subjective terms. However, the customer is often not the only important stakeholder, and one has to address all the concerns of key stakeholders to deliver a successful service. The matrix of concerns, constraints, opportunities, etc. is a hazy mesh that takes patience, knowledge, imagination and tact to navigate.

Oh, for the halcyon rookie days, of operating below the radar of key stakeholders, architectural governance, and budget holders! For certainty and technical purity, for the stream of tactical solutions, and the buzz/hit of seeing “it” work! I still remember one moment of amazement, when a visiting consultant pointed out to me that the collection of MVPs I had laboured over for the last two years was actually a mini ERP. Those were some memorable years, in London, Edinburgh and Southampton. It was a lot of adrenalin, noise, banter, late nights, early mornings, intense work and much fun.

I now work mainly in the integration & solution architect scopes, synthesising services from business, architectural and technical inputs. Constantly scanning for opportunities to drive innovation, efficiency and competitiveness, while also reducing complexity, cost and TTM. In addition to the requisite technical and business knowledge, my role involves tact++, listening++, politics, psychology, emotional intelligence, and social strategy. It is not a straight line, there are no easy answers, there is little fizz, and you do not enjoy a fiat in any context.

Neither do I ever deliver anything alone. I always work with others, no one works for me, but rather with me: hierarchies can oft-times be an impediment. I depend on others to implement, provide or revise funding, change scope, grant a dispensation, review a requirement, move a date, or compromise in one way or another, to advance the delivery of solutions, and value to our customers. Customers are the ones that pay for our activity, and it is key that we understand their perception of value in order to consistently deliver services that they are willing to pay for.

The customer is king! But KYC cannot always be gleaned from the organogram or reading through the requirements. What may be right for one customer with a tight and close market window might be an unacceptable hack for another. The solution that was gladly embraced by customer-A in Q3 of this financial year, when revenue was trending up, could be flatly rejected in Q1 when corrections reveal shortfalls. Over time, one does gain knowledge and one does build trust, and all this widens the scope of influence. However, with such a wide matrix of interdependencies, the role is always challenging and unpredictable.

All said and done, it is still very interesting and rewarding work. It is a journey of discovery, of self and others; of building relationships and trust; of learning from failures and successes; of growing in patience and perception, and of seeing the sometimes hard-to-perceive positive impact of changes one helped to nurture. And “Yes, it is a good life, ‘Henry’“.

Kubernetes-to-AWS Security Bridge

AWS Kubernetes RBAC Bridge

Background

Security in an interconnected, always-on (24*7), virtualised, digital world is important. As more of our IT infrastructure moves to the Cloud, proactively seeking and blocking emerging security gaps becomes a continuous activity (BAU).

AWS and Kubernetes are leaders in the new paradigm of abstracted infrastructure – Cloud, datacentre on on-premise. Both have their own evolving security arrangements. For role-based access control (RBAC), AWS uses the IAM primarily, while Kubernetes (K8s) uses a combination of Roles and Role Bindings. The primary RBAC intersection point between these two has been the node/virtual-machine (EC2).

The Problem

In a simple world, privileges assigned to the underlying AWS node are inherited by the K8s Pods running on the node. This works perfectly when there is a one-to-one mapping between the client of K8s and the consumer of the AWS node. Specifically; the same entity owns the K8s cluster and the AWS node on which it runs. Security is intact, irrespective of the number of K8s Pods on the AWS node. However, misalignment occurs when K8s shares the same node among two or more clients – often referred to as multi-tenant mode. A potential for a security breach emerges.

Imagine a scenario in which there are three K8s Pods (A, B & C) running on a single AWS node. Each Pod runs a service that belongs to a different customer/client. Pod A belongs to client-A, Pod B belongs to client-B and Pod-C belongs to client-C. Files needed by each client are stored on S3 buckets in AWS, and each client has responsibility to arrange for their own S3 bucket. However, client-C is the only one that has managed to provision an S3 bucket at the time of deployment. Ordinarily, Pod A and B should never access the resource(s) provided strictly for Pod C. But if they do, nothing stops them! The diagram below provides a useful illustration.

K8S-AWS-RBAC_Quandary

Historically, in IAM, access privileges to the resource for Pod C will have been given to the node hosting Pods A, B and C. The EC2 node would have an Instance Profile defined, and a Role will be attached to the Instance Profile, giving it those privileges. The unexpected consequence however is that Pods A and B also inherit the privilege from the host node. Pod C’s resources would therefore be accessible to any other Pod (client) running on that node. This obviously is not acceptable for a multi-tenant K8s cluster.

Solutions

The great thing about the Open Source community is that problems are attacked, and often solved, almost as soon as they are articulated. Two open source products emerged to close this security gap: Kube2IAM (2016) and KIAM (2017). Some time later, AWS introduced a solution; “IAM for Service Accounts”. However, the AWS solution only works with their EKS service. All three solutions make it possible to control access from K8s Pods to AWS resources.

I will not discuss the AWS solution as it is proprietary and closely tied to their EKS offering. Neither will I examine KIAM as the solution has been abandoned by the developers. This leaves us with the forerunner: Kube2IAM. Kube2IAM deploys a K8s DaemonSet in the K8s cluster. By default, one Kube2IAM Pod is deployed to each worker node in the cluster. The Kube2IAM instance running on each node intercepts requests to the AWS metadata service URL (http://169.254.169.254). It then provides a response according the the IAM role assignments, as well as the annotations on the Pod calling the service. The diagram below provides a useful illustration.

AWS Kubernetes RBAC Bridge

With this simple solution by Kube2IAM, the exclusive role assignment to Pod C is respected by K8s. Deliberate or inadvertent requests by Pod A or B are blocked by Kube2IAM.

Here is how it works. When a Pod makes a request for AWS resources, it will make a call to the AWS metadata service URL. Kube2IAM hijacks the call (iptables reroute) and performs an inspection to see what the appropriate response should be. It checks if there are any appropriate RBAC annotations on the Pod making the request. If there are none, Kube2IAM serves up the default privilege set. These will be the privileges defined for the EC2 Instance Profile. However, if the Pod has a role annotation, it will be given the privileges defined in the matching AWS role.

Hands-on

In the example that follows, we will deploy two Pods; one with annotations (annotated) and another without (vanilla). We will use two AWS roles. The read-only role will have access to one S3 bucket only. The other read+write role will have read access to 2 buckets and read+write access to one bucket. The read-only role will be attached to the EC2 Instance Profile for the K8s worker node. The read+write role be standalone, but it will be extended to trust the read-only role. This sets the stage for Kube2IAM to discriminate between requests, giving read and/or write access to our Pods, as appropriate. In our example, the annotated Pod will be able to write one bucket and read two buckets, while the vanilla Pod will only be able to read one bucket.

The implementation artefacts can be downloaded from GitHub (use this link). I have put together what I think is a simple, and perhaps more explicit set of instructions below. Follow them step-by-step and you should end up with a working RBAC bridge using Kube2IAM. I guess one could write a script that automates all of these steps, but that is a task for another day, or perhaps someone else.

Process

  1. Create a policy (nettech-s3-read-only); use the file nettech-ec2-instance-profile.json for the policy definition/contents
  2. Create a role (nettech-s3-read-only); the role should refer to the policy in step #1
  3. Create an EC2 instance profile (nettech-instance-profile) for the AWS node(s); the instance profile should refer to the role you defined in step #2, forming a chain:
    nettech-instance-profile==>nettech-s3-read-only(role)==>nettech-s3-read-only(policy).
      Use the following aws-cli commands:
      aws iam create-instance-profile –instance-profile-name nettech-instance-profile
      aws iam add-role-to-instance-profile –instance-profile-name nettech-instance-profile –role-name nettech-s3-read-only
  4. Create a second read+write S3 policy and role (nettech-s3-read-write). Use the file nettech-s3-read-write.json for the policy definition/contents
  5. Extend the trust relationship on the read+write S3 role such that it can be assumed by the read-only role, forming a link:
    nettech-s3-read-write(role)<==trust==nettech-s3-read-only(role).
      In IAM console, select the read+write role
      Select the “Trust relationships” tab, and then click on the “Edit trust relationships” button
      In the new window that opens, add the contents of the file nettech-s3-read-write-trust-relationship.json to the existing definition/contents
      Make sure to update the AWS account Id (01234567890) to your own
      Click on “Update Trust Policy” to save your changes
  6. Deploy or assign a worker node in your K8s (Rancher/Kops/..) cluster
  7. Configure or update the worker node to reference the EC2 Instance Profile (nettech-instance-profile) from step #3
      aws ec2 associate-iam-instance-profile –iam-instance-profile nettech-instance-profile –instance-id xxxxxxxxxx # replace xxxx with your instance Id, or use the AWS GUI to attach it
  8. Deploy Nginx vanilla and annotated (K8s Deployments). Use the file nginx-deployment.yaml from Rancher UI or kubectl on the command line
  9. Install aws-cli in each of the Nginx instances – Use the following commands (Linux):
      apt update
      apt install curl -y
      apt install unzip -y
      curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
      unzip ./awscliv2.zip
      ./aws/install
  10. Verify that the host node has read access to the “nettech-helm-test” bucket, according to the EC2 profile from step #3. Connect to the host node (via Rancher UI or SSH) and run the aws s3 ls command.
      aws s3 ls nettech-helm-test
  11. Verify that both Pods have read access to the “nettech-helm-test” bucket. Connect to each Pod (via Rancher UI or kubectl) and run an aws s3 ls
      aws s3 ls nettech-helm-test
  12. Create/deploy ClusterRole & ClusterRoleBinding for the service account to be used by Kube2IAM. Use the file clusterRoleAndBinding.yaml
  13. Deploy Kube2IAM (K8s DaemonSet), with debugging enabled. Use the file kube2IAM-daemonSet.yaml
  14. Connect to the host node and access the command line. Check that only one IPTables rule exists on the worker node (for AWS metadata IP). Delete any duplicates to avoid confusing errors. This may happen if you redeploy the Kube2IAM Daemonset.
      sudo iptables -t nat -S PREROUTING | grep 169.254.169.254 # list all entries
      sudo iptables -t nat -D PREROUTING -d 169.254.169.254/32 -i docker0 -p tcp -m tcp –dport 80 -j DNAT –to-destination 10.43.0.1:8282 # delete any duplicates
      NB: (docker0) is the network interface, (10.43.0.1) is the IP address of the node/host, and (8282) is the Kube2IAM port
  15. Test the Nginx instances again
    • Verify that the host node still only has read access to “nettech-helm-test”, as defined as a default in the EC2 Profile role (nettech-s3-read-only)
    • Verify that the vanilla Nginx Deployment still only has read access to “nettech-helm-test”, as defined as a default in the EC2 Profile role (nettech-s3-read-only)
    • Verify that the annotated Nginx Deployment now has read access to “lanre.k8s.dev” and “nettech-helm-test” as well as read+write access to “lanre.k8s.dev”

Conclusion

An RBAC bridge of some sort is a necessity for all multi-tenant K8s clusters running on virtualised infrastructure such as AWS, Azure, GCP and others. Kube2IAM provides an effective solution for the AWS platform. This article identifies the issue that Kube2IAM resolves and shows a very simple, sandbox implementation. The article should serve as quick-start guide that is easy to grasp and quick to implement.

We live in a rapidly evolving technology environment. Kube2IAM has set a very sound foundation, but as always, there is always room for improvement; and I say that with all humility and respect for the developers. KIAM came up with a cacheing service to reduce latency and improve scalability, unfortunately, that solution is no longer being evolved. One would like to see similar functionality in Kube2IAM. One other improvement would be to move the annotations out of the Pod and into K8s roles. The preference being roles defined outside the namespace of the beneficiary Pod. This will reduce the attack surface for malicious code that might attempt a brute-force attack to find AWS roles that can be exploited.

Many thanks to Jerome Touffe-Blin @jtblin and his team for creating this ever-so-useful open-souce utility.



Oyewole, Olanrewaju J (Mr.)
Internet Technologies Ltd.
lanre@net-technologies.com
www.net-technologies.com