This article is NOT generated by LLMs or AI tools.
Years ago, when I first read The Master and His Emissary by Iain McGilchrist, I wasn’t expecting it to totally reshape my thinking about how we define data, how we store it, how we retrieve and query it and so on. Perhaps I had forgotten that I was (still am) a software engineer. After all I have been taking all my learnings from arts, sports, and other non engineering fields to engineering and benefitting from this knowledge transfer across domains quite a bit. This was a time when I had already published a couple of books and had worked on a few others as technical reviewer/editor and my blogging days were seemingly coming to an end. Not to mention my private life was also getting busier and busier, so there was not much time to blog either. For this and many other reasons, I wasn’t really getting enough time to blog about how my thinking about data was changed and share my learnings. Which brings me to the title of this post, and graphs in general.
I don’t want to compress down Iain McGilchrist’s book into a single sentence or paragraph, especially since one of the major arguments of the book is literally about tendencies like that, but to simplify and convey the point I have to boil down my takeaway to one main topic. Context and its importance.
Abstracting away the context from anything we are trying to build, or fix, or heal, or change and so on, allows us to have a simpler and manageable view of it, which in turn leads to more and better abstractions and it leads to easier solutions for much complex problems. We do this in almost every field we deal with. For example, if we have pain, we take painkillers. Look around you and you would see this all the way from medicine, to engineering and politics and so on. Get rid of the context (to the extent that is possible) and the problem you are trying to solve becomes much easier and manageable.
However, this approach has limitations, and in recent years its limitations have become more and more obvious. We fixate on pinpointing, compressing, abstracting and overall removing context to a degree that what remains is an empty shell of the thing we were dealing with. Again, this has many manifestations but I will look at it from the perspective of a software engineer. And this brings me back to data and graphs.
For a long time, I never even questioned the simple tabular nature of data I saw around me. It almost puts my mind at ease to see an excel sheet, or a table in a SQL database. Each row representing a person, a company, an object or whatever else. Each column represnting a name, or age, or area, or type or whatever else. However, that is not how we understand data. We do not think about any entity the way its stored in a tabular format. Granted, it simplifies the storage and retrieval, but it is not what we imagine when we see those rows and columns. A person, is an interconnected being that yes has some properties (age, height, …) but it also has friends, connections to other entities, loves, hates, misses and so on. You might be thinking, all of that can be stored in tables, and that is true. Until, we hit the threshold of connections and links being too deep. (Ask anyone who understands SQL about deep and nested JOIN statements)
To overcome this, graphs represent an incredibly elegant structure. And they are much closer to how we see and understand the world around us. As I started thinking about this, I realised this must be something that others have thought of (of course they had) and thus I stepped into the world of graph databases, nodes/vertices and relations/edges, and ontologies and more and more to be learned.
Over the years I have done a huge amount of work on graphs and unfortunately I never thought of them as worth sharing or seeing any interest for it in my circles (perhaps it has to do with the fields I have worked in) but as LLMs and AI tools took over, I suddenly realised that context is becoming more and more important, and not to mention expensive. LLMs have limited context windows (yes, even millions of tokens is a very low bar for real context in real life problems) and all sorts of methods (such as RAG) are being tried to migitage this.
And finally, to sum it up, I am not sure if it is possible or necessary to include all the context from reality in order to have a holistic view of what we are trying to understand, fix, heal and so on. But after working with graphs for so long, I am certain they represent a much closer structure to reality than tables and other strcutures we have been dealing with for so on. As a result, I will be sharing some of what I have been doing with graphs moving forward. Stay tuned and enjoy!