Results of 2024 for me and the blog
Blog statistics for 2024. Accurate statistics start around March, as the blog used the old analytics service from January to mid-February.
Let me share what I was up to in 2024, how my plans for the outgoing year [ru] turned out, and what I plan for the year ahead.
AI notes 2024: Prognosis
I continue my notes on AI at the end of 2024.
In the previous posts we discussed three theses:
- By analyzing the decisions of major AI developers, such as OpenAI or Google, we can make fairly accurate assumptions about the state of the AI industry.
- All current progress is based on a single base technology — generative knowledge bases, which are large probabilistic models.
- The development of neural networks, a.k.a. generative knowledge bases, is reaching a plateau; their further advancement is likely to be incremental/evolutionary rather than explosive/revolutionary.
Based on these theses, we can finally talk about the most hyped, most exciting topic: how long will the current progress continue? Will we reach the singularity in 2025, or will everything remain the same? When will our god of metal and silicon finally appear?
In 2023, I already published a forecast on artificial intelligence [ru]. It is still valid — take a look. In it, I spent more time describing what to expect. This text is more about what not to expect. So, it turns out that I outlined two boundaries of the possible, and the truth should be somewhere in between.
Impressions after reading Harry Potter
The classic cover of the first book.
As part of my immersion in modern world culture and integration into the Hamburg community (the city is a major fan hub), I read all seven Harry Potter books. I even specifically got the English edition to read in British English, not American (although Reddit says the difference between them is small).
Impressions are mixed.
On the one hand, the story is interesting, written in good lively language, the first books are read with pleasure, the last ones without significant suffering.
On the other hand, I expected more. After reading, I can't reconcile the quality of the books with the level of their popularity.
On the third hand, I believe Joanne Rowling could have done it better :-) Not that it's written poorly — it's written well — but too one-sided, too narrow, I would say. Especially the last books.
Let me tell you a little more about the books.
AI notes 2024: The current state
I continue my notes on AI at the end of 2024.
- Industry transparency
- Generative knowledge base
- The current state
- Prognosis
In the previous posts, we discussed two theses:
- By analyzing the decisions of major AI developers, such as OpenAI or Google, we can make fairly accurate assumptions about the state of the AI industry.
- All current progress is based on a single base technology — generative knowledge bases, which are large probabilistic models.
Based on these theses, let's look at the current state of the industry.
Fun case of speeding up data retrieval from PostgreSQL with Psycopg
Speed of data retrieval from the PostgreSQL for each of the optimizations. In percentages of the base implementation speed. Note how the number of retrieved rows has little effect on execution time.
Once in a year or two, I have to remember that Python, umm... is not C++. It usually happens suddenly, like this time.
After a thoughtful analysis I got tired of waiting 10 seconds for news to load on feeds.fun, so I rolled up my sleeves and started optimizing. I almost jumped into an epic refactoring, but remembered just in time: measure first, then cut. In this case, the advice is literal — I took a profiler — py-spy — and checked what exactly was causing the slowdown
It turned out that the problem is not in the entire logic, but in a very specific place where the code extracts ~100000 rows from a PostgreSQL table, plus-minus 20000. The indexes are in place, I ran the tests with a database on a RAM disk, so everything should have been fine from the database side.
Don't be surprised by such a large number of rows:
- Firstly, I have a substantial news flow.
- Secondly, the reader currently assigns about 100 tags for each news item.
Armed with py-spy and the sources of psycopg, I went through three stages of optimization, reducing the function execution time approximately 4x solely by changing the format of the requested columns in SELECT query and the result processing code.
In the following text, I will tell you about the sequence of curious discoveries I made in the process.
Attention!
This post is not a study of Psycopg or Python performance, but a description of a specific experience with a specific task and specific data.
It would be incorrect to judge the performance of Psycopg, or Python in general, based on a single specific case.