Geoffrey Ellison

Meta-Aware Means Self-Aware

Meta-Aware… is that self-awareness? A self-aware system? Like Skynet, the evil take-over-the-world-and-kill-all-humans system, that dreamed up the terminators? No, not quite.

In this context, I’m talking about a system that is not explicitly programmed for every possible situation. Rather, it will look inside itself, in its own data, to see how a particular situation is to be handled… in terms of what it should expect, what processing it needs to do, where and how to return the output. The what-to-do-next instructions are not explicit, as is usually the case in programmed systems. The meta-aware system doesn’t “know” how it will react in a given situation until the situation arises.

I know what you are thinking: what’s the point? Or: is it even possible? Let me address these one by one:

What’s The Point?

The point is simply that you end up with a system of such flexibility, such adaptiveness and resilience, that a business can benefit mightily from using such a system as its core business engine. And yet, some of the most exciting benefits are felt during the development process itself. Let me highlight some of the major advantages:

True Delegation of Functionality: Different programmers (of varying prowess) can work on (and be responsible for) different areas of the system at the same time. This brings source control to a whole new level. Like Monty Python’s famous “funniest joke of all time”, you needn’t worry that any one person will see too much of the system at once (requiring hospitalisation). This also neatly solves the MVC problem. Not only does it keep the front-end (UI) in the hands of the front-end developers, but it also keeps the program code relating to front-end functionality (for example, PHP code to build an HTML list) in the appropriate hands also. And the same is true for the back-end, communications layer, database layer, business objects, and so on.

Preservation of Intellectual Property: The delegation of functionality above means that each developer only sees the parts they are working on, or at most, their library set. Yet the hierarchy of the system is known to all developers, and by design, each unit and library is fully documented. It has to be to form part of the meta-structure. All functionality is accessible, yet only permitted code is exposed. This preserves your intellectual capital investment from theft, but unlike other systems designed for this purpose, it manages to do this without reducing productivity or hampering developers’ creative flow.

Unparalleled Flexibility: This is not just an idle boast. The hallmark of any powerful system is that it can be made to grow beyond the uses that were envisioned when it was created. For example, cell phones were intended to be telephones. But by becoming computers, on which any given software can be loaded, the door is opened for them to be just about anything. So it is with a meta-aware system. If an unforeseen need arrives one day, you don’t need to put your whole team on it. Just add a new library (even that may not be necessary – a few definitions may do the trick) and suddenly your system is handling things that weren’t even conceived when it was designed. The same applies if a new protocol comes along, or if your suppliers or customers suddenly start insisting you use a totally new mechanism to communicate with their systems. No problem. A new (minor) library, or a few lines of service definition, and it is all done.

Transparency In Development: During development, the path to a meta-aware system is quite different from traditional development. The system will “grow,” almost organically, with each piece being completed as needed. Functional units may be delegated entirely to a team or an individual, and broken down further into classes, objects, procedures or functions. At any time, a quick glance will tell you exactly how each unit is coming along, and it is possible to view progress at the highest level (the entire project), or drill down to the smallest functional item. A well-built meta-aware system will be able to provide all this information itself, without recourse to external project or reporting tools.

Budget Enforcement: The core of a meta-aware system will not take long to build, and after that is done, you can settle back and proceed with adding branches, stems and leaves as the budget allows. This is great when development is funded out of cashflow, as there is always something for your developers to do, and they can be concentrated on areas of greatest need, but also moved to different areas of functionality when required. It is also very easy to spin up new teams or individuals quickly, and they can be spun down just as fast.

Potential To Reduce Time To Market: It is often said of software development that “Nine women cannot make a baby in one month.” And that is true, with traditional software development. However, because of the way a meta-aware system is structured, it becomes feasible to proceed on several vectors simultaneously. This comes into its own whether you are outsourcing all development or just overflow to cheaper communities, in India, Asia, etc., communication difficulties (which are often project-killers) tend to recede as functional units are broken down into basic building blocks. This also has the pleasant side effect of reducing the number of senior developers needed, as the actual execution will not require very strong big-picture skills.

Seamless Integration, Even With Legacy Systems: It’s great when you get to build a green-fields development, where you get to architect the system itself, as well as all other systems it may talk to to, including billing, communications, reporting, and so on. But this almost never happens in reality. Even in a green-fields project, it’s not advisable to re-invent the wheel, and an architecture that calls for the building of components that are already commercially available is probably not one you want to put your money into. In practice, even the most sequestered system is an integration exercise, talking to all kind of external systems. Usually those systems make up the product and service offerings that your business is depending on to provide to clients. They may be provided by you, or provided by a vendor, managed directly or managed in the cloud. A meta-aware system will have simple data collections describing those systems, with libraries to carry out each method of communication. Base methods can be developed (for example SSH session support) and then extended with platform-specific features (such as script calls in a SSH session). It makes no difference how diverse the systems are. All systems have inputs and outputs, and a meta-aware system will know just what to say.

Powerful Automation Capabilities: Wherever possible, you want to be able to turn a customer action into a result without any human intervention. For example, if an existing customer wants to upgrade or add a product, the events that flow from their action should result in the request being fulfilled (or denied politely, with an explanation), without human intervention. This is the magic key that unlocks a business’s ability to SCALE beyond what its staff can provide. In essence it gives you the ability to service customers while you sleep. A meta-aware system will know what services are available to each customer, and exactly what is needed to provision those services.

Integrate New Systems Without Code: Once a meta-aware system has had a few methods defined, integrating with a new system is simply a matter of storing a few definitions: namely, the types of actions available on a service, how to call those actions, what data to send, how to format it, and what to expect in return. Even if you are integrating with a system via a method that has never been defined before, you can simply extend an existing method. For example, a SOAP call is an extended subset of an HTTP call. With a well-built meta-aware system, integration becomes trivially easy.

This All Sounds Great, But Is It Even Possible?

Yes it is. I know because I’ve built one. It was a complete management system for an Internet Service Provider, covering the customer UI, the administrative UI, integration with (a broad range of) suppliers’ systems, telco integration, automatic provisioning, reporting, billing, business intelligence, and more. Several years on, I have no doubt that it is being used for a whole lot more than I could possibly have envisaged, and yet I know that each new capability needed only a few lines of code to make it happen.

So How Do I Build A Meta-Aware System?

It is surprisingly simple. Your core should look something like this:

I’ll explain the above by means of an example: an online, web-based email system (such as Gmail, Hotmail, etc). Let’s start with Services. A Service corresponds to the physical manifestation of an aspect of an entity. To express that mouthful in terms of our example, the service is a product, or product option, such as the mailbox itself, an anti-spam product, extra storage, and so on. Although our system controls this service, it is likely to be externally-hosted, with its own system and data. It may even be provided by a external vendor. The collection of Service meta-data is what tells the system what services are available, and how to manipulate them.  The Actions collection will typically express what the entity is capable of doing with Services, for example, AddNew, CheckAvailable, ChangeExisting, DeleteExisting, and such. The Methods collection informs the system how to communicate with the external infrastructure that will provision, modify or delete the Service. For example, some services expect to talk via a SOAP or RESTful API, whereas some may be command-line-based. Some may return an “OK” response in plain text, whereas some may provide an ID that links to a response table. And finally, the Fields collection describe to the system what kind of data it should be sending when executing each Action, providing a translation between the fields of the internal entity data in our own datastore, and the fields that the Service expects to receive. It also defines any conversions that may be needed to make the data acceptable to the service.

How Does This All Fit Together?

Let’s consider an example sequence: a customer of our webmail system wants extra storage. The customer starts the process by visiting a web page (or application) that lists available services. If our meta-aware system thinks this customer is entitled to be offered the extra storage product, it will be displayed in the list of options. So our customer clicks on “Extra Storage” to start the ball rolling. In our system’s Fields collection, we are aware that certain information is needed in order to create an extra storage account. So we get some of the information from the customer’s record in our database, and some of the information (such as how much storage is needed) we get directly from the customer. Now armed with everything we need, the system sees (in its Methods collection) that the storage server should be spoken to using the SOAP protocol, with certain passwords for authentication, so it makes a SOAP call to the specified URL, calling the function to add a new storage account (which comes from the AddNew action in the Action collection). It knows what fields to send, and it knows what to expect in return. The result is analysed and the customer is informed of its success or failure.

That is the sequence through the core of our meta-aware system. In addition, you would also have an automation chain that ties into a billing system, product management system, and others. You may also have service groups, as it is quite likely that a real-world service is made up of several Services defined in our collection. For example, providing an email address means providing an alias, mailbox, POP3 access, IMAP access, and so on. Each one of these would be a part of an automation chain that links these services together.

In A Nutshell…

Now you know that a meta-aware system is equipped to decide what services it can offer, and to who, and it knows how to provision or manage those services, what to send, and what to expect in return. A meta-aware system can carry these actions through to conclusion without any human intervention. During its development, a meta-aware system can also simplify the development process, making it easier to achieve deadlines, share work, secure intellectual property, and bring teams up to speed. A meta-aware system can seamlessly communicate with any system out there, even cumbersome legacy systems, to do its bidding. A meta-aware system is the focal point of a mesh of disparate systems, all made to work in unison and controlled from a central point.

Maybe it is like Skynet after all…