Making Obsidian Really Work

While documenting with Obsidian and Github Markdown, following tips have been most useful:

Headings...

are H1 to H6, use # sign the number of times you want the heading type e.g. #### is for H4

Linking... 

• to a document: type `[[`, a list of documents you can link to appears
• to a heading in another document: type `[[`, a list of documents you can link to appears, select the document to list, the use `#` sign, the headings that can be linked appears, just click the heading to link to
• to a text block in another document: type `[[`, a list of documents you can link to appears, select the document to link to, the use `^` sign, the text blocks that can be linked appears, just click the text block to link to
• to a heading in another document: type `[[`, a list of documents you can link to appears, select the document that is currently being edited, the use `#` sign, the headings that can be linked appears, just click the heading to link to
• to a text block within the document: similar to anchor tags, type `[[`, a list of documents you can link to appears, select the document that you are currently editing, list of text blocks in the document appears, select the text block to link to

• to an image

            ![[AbsolutePathOfImage]]

• to an image with an alternate name to display

            

• to an image and resize

            ![[AbsolutePathOfImage|500]]

Using tags

#FollowedByNoSpace is a tag, used in search and in generating automated knowledge graph management

Arrows

• Up arrow (↑): ↑ 
• Down arrow (↓): ↓ 
• Left arrow (←): ← 
• Right arrow (→): → 
• Double headed arrow (↔): ↔

Colouring the text...

• View developer console --> (on Mac: ⌘ + option + i)
• And then edit CSS elements when it becomes necessary

Superscript or Subscript

• Prefer to use latex formats
• Superscript ^
• Subscript _

OR

• In the developer console, use html tags 
• ^{some text}
• _{some other test}

Managing Technical Debt

Introduction

The gap between the current state of a software system e.g. inefficient code, design flaws, missing documentation etc. is usually called technical debt. It represents the future cost that will be incurred when addressing such shortcomings e.g. as fixing bugs, refactoring code, improving documentation, making the system more secure and scalable.

Technical debt, incurred intentionally or unintentionally is not necessarily a bad thing in all cases. In some situations, technical debt results from strategic decisions made to meet urgent business needs or to gain a short term but competitive advantage. However, it must be managed and paid down over time to prevent any overwhelming hinderances to future development. 

Technical Debt Management Framework

To effectively manage technical debt, it must be recognised, documented, and tracked as part of the software development process. This helps make informed decisions about when and how to address the debt by balancing short-term goals with long-term sustainability and quality.

The industry recognises and categorises a number of forms of technical debt. However, the following are of most interest for an investment banking enterprise:

Difficulty column gives an indication of effort that may be invested in managing and repaying a given category of the debt. As an example, the commonly identified 'Architecture and Design Debt'  is quite difficult to identify and measure in hindsight, because the decisions are soundly supported by business needs. Usually 'Architecture and Design Debt' is also closely related to other debts e.g. 'Integration Debt' and thus very difficult to even monitor and track. This debt is also very difficult to repay since architecture and design are rarely revisited without motivation and certainly not only with just the intent to repay by remediating and refactoring code.

The framework considers 'Architecture and Design Debt' as an umbrella encompassing 'Code Debt', 'Performance Debt' and others as remediation sub-categories (add a column) with the following considerations:

• Performance Debt: Monitor performance using APM tools, identify performance bottlenecks | Set performance benchmarks, track metrics | Continuously monitor performance, allocate resources to address issues | Variability of workloads, difficulty in isolating performance causes | 
• Code Debt: Evaluate code quality using metrics such as cyclomatic complexity and code duplication | Maintain code documentation | Conduct code reviews, implement automated code quality checks | Complexity of large codebases, rapidly evolving code

One may want to add 'Skills Debt' to the framework. It is medium difficulty, can be more scientifically measured & remediated and helps to repay other types of technical debts by a well managed and self motivated team.

The framework as well identifies challenges to expect and lists some generic steps that could be taken to alleviate the issues while actively managing and repaying technical debt.

Scratching the OLAP surface

Introduction

Long ago, business transactions were saved to relational databases and few ad-hoc decision support queries ran on the same database. As data size grew and time sensitivity of business decisions reports became apparent (e.g. fetching monthly sales reports), running complex and expensive SQL queries on the same database impacted saving business transactions to the DB.

Running complex and expensive SQL queries was moved to a purpose-built database called data warehouses (DWH). Data was sent from multiple business operational systems and integrated into central storage - the DWH, a single source of truth of business data.

Data warehouses were built on MPP principles (massively parallel processing) i.e. compute and storage were coupled to benefit from the data locality during query execution. To speed up the complex and expensive queries ETL pipelines pre-joined and normalise data before loading into DWHs. The storage footprint reduced and queries ran faster with tables laid out in two main strategies viz. star and snowflake schemas.

Finally business teams could run complex queries that filter and aggregate billions of records and analyse large volumes of data from different perspectives with help of BI tools. Thus business transaction processing separated from decision supporting analytical processing - former became OLTP and the later was the advent of OLAP.

The advent of OLAP

A data warehouse remains to be a large relational database storing data in a collection of tables unless the following additional layers make it into a multidimensional database:

• Data is modelled by several dimensions to reduced storage footprint, improving query times e.g. tables are laid out in two main strategies viz. star and snowflake schemas.
• ETL pipelines ensure pre-joined and normalised data is loaded into DWH
• Complex queries with several joins and higher level aggregates are managed as materialised views
• Dedicated UI to run complicated queries across several data dimensions

Such multidimensional database deployed on OLAP server, providing dedicated client interface to run complicated queries, visualise data and perform analytical operations across several data dimensions for generating business intelligence insights came to be called an OLAP Cube.

Key differences in OLAP and OLTP

	OLTP Applications...	OLAP Systems...
Purpose	are used to run the business	are used to understand the business
Purpose	handle large volumes of transactional data from multiple users e.g. online hotel bookings, mobile banking transactions, e-commerce purchases	quickly process large amounts of data for in-depth data analysis across multiple dimensions for decision-making
Data sources	data is created by users as they complete business transactions	pull data from OLTP databases via an ETL pipeline to provide insights such as analysing ATM activity and performance over time
Response times	milliseconds	range from a second to several hours
Data storage capacity	usually have modest data storage requirements, as historical transaction data is archived	require massive amounts of data storage capacity, a modern cloud data warehouse may accommodate these easily
Intended Users	are customer-facing and designed for use by frontline workers such as store clerks and hotel reservation specialists as well as online shoppers	are business-facing and are used by data scientists, analysts, and business users such as team leads or executives who access data using analytics dashboards

Types of OLAP Systems

Depending on specific analytical needs of an organisation, the amount and complexity of data to handle, the required query response times, and the kind of analysis and reporting required determines which of the following OLAP systems be used:

Relational OLAP (ROLAP) systems... 

1. facilitate multidimensional data analysis with high data efficiency
2. SQLs retrieve and analyse data from relational tables of data warehouse
3. are highly scalable and can handle large amounts of data
4. have slower query response times and do not support complex calculations

Multidimensional OLAP (MOLAP) systems...

1. are fast for multidimensional analysis and running complex calculations & aggregations
2. store data in a multidimensional cube format, where each dimension represents a different attribute of the data e.g. time, geography, or product
3. require extensive data preprocessing (as data is stored in multidimensional cubes)
4. can handle limited data and are not as scalable as ROLAP systems

Hybrid OLAP (HOLAP) systems...

1. combine the strengths of MOLAP and ROLAP systems
2. store summary data in multidimensional cubes while detailed business data is stored in relational database, thus also improving data relevance
3. provide fast data access for high speed querying and handle high volumes of data

Data storage strategies in OLAP systems

Star Schema...

• is multidimensional data model, used in ROLAP systems
• organises data into a central fact table surrounded by dimension tables
• fact table contains measures being analysed i.e. quantitative data like sales revenue, quantity sold, profit margin etc.
• dimension tables contain descriptive data that provide context for the measures e.g. time, geography, product information, etc.
• each dimension table is joined to the fact table through a primary key-foreign key relationship
• is popular as it is easy for business analysts and end users to understand and navigate through different levels of data

Snowflake Schema...

• is ROLAP data model
• organises data in a central fact table and normalised dimension tables (i.e. multiple broken but related tables)
• normalises dimension tables to reduce data redundancy and improve data consistency
• has more tables and relationships resulting in hard to understand, complex and slow queries
• improves query performance and reduces storage requirements by eliminating redundant data

Fact Constellation (aka Galaxy) Schema...

• contains multiple fact tables, each with its own set of dimension tables containing descriptive data
• the fact tables have shared dimensions which links the fact tables and hence allow for even more complex queries and analyses
• each fact table represents a different business process or measure e.g. sales or customer satisfaction
• provides more flexibility in querying and analysing data, as users can analyse multiple business processes or metrics at the same time
• is harder to use than the star or snowflake schema

Improving performance of OLAP systems

Pre-aggregating data for faster access

• pre-calculating and storing summary data in OLAP cubes e.g. totals, averages, etc.
• combining data at different levels of granularity for readily providing high-level overviews

Caching data for quick retrieval

• repetitive queries and frequently accessed data & query results are stored in memory

Indexing on specific columns and dimensions

• helps to quickly locate the required data rows without scanning entire storage system

Partitioning into smaller units

• to optimise performace database engineers divide large tables or cubes into smaller, more manageable parts based on a partitioning key thus reducing the amount of data that needs to be scanned for each query

Parallel processing

• a query is divided into parts, known as tasks, and distributed across multiple processors or cores. Each processor is assigned tasks simultaneously, allowing the query to be executed much faster than if it were processed sequentially on a single processor.

Materialized Views

• pre-calculated views store results of complex queries as physical tables

Hardware and infrastructure

scalable infrastructure e.g. fast processors, large memory, high-speed storage etc., facilitate data discovery, unlimited report viewing, and complex analytical calculations. Cloud-based vendors for data analysis are now a default choice. They simplify integration, are reliable, easy to scale, and more affordable than on-premise data infrastructures.

Data preparation... 

Data gathering, storing and cleaning is done via two data integration methods viz. ETL and ELT.

ETL (Extract, Transform, Load) is a predefined sequence for extracting data from sources, transforming it to meet the target system's requirements, and loading it into a target data warehouse. It is complex, time-consuming and requires upfront planning for the data to be correctly transformed and loaded into the target system. 

ELT (Extract, Load and Transform) involves extracting data from data sources, loading it into target data warehouse or data lake, and then transforming it to meet the target system's requirements. 

Differences between ELT and ETL

Unlike ETL, ELT does not require a predefined sequence of steps. The extracted data is loaded into the target system as quickly as possible, and then the data transformation process is applied to it in-place.

ELT solutions are usually applied to modern cloud-based data warehouses that allow for massive parallel processing. So, ELT solutions can process large amounts of data much faster than traditional ETL solutions. Also, ELT solutions are more flexible than ETL solutions, as they allow for data transformation to be performed on the destination system in-place.

Once data has been loaded into a cloud data warehouse, engineers and analysts use modern data stack to prepare data for analysis. 

OLAP Operations

• OLAP systems use a specific SQL language called MDX or Multidimensional Expressions
• Also support standard SQL queries to perform OLAP analysis

Some standard multidimensional OLAP operations are:

Slice and Dice

• Slicing: dividing one dimension within the cube into a separate table, enabling low-level and isolated analysis of a data set
• Dicing: is dividing two or more dimensions within a cube to generate a separate cube

Drill down and Roll-up

• drill down: is move from high-level data to view lower-level information
• roll up: is move from detailed data to less detailed data or summarised information

Pivot

• is to rotate data from rows to columns or from columns to rows, enabling multidimensional analysis from different perspectives and data comparisons across dimensions

Drill-through

• creating data points to access detailed information faster. When users click on a data point in a summary, they are shown the underlying data that make up the summary

Drill-across

• using a common dimension shared by different data sources to enable data analysis across multiple unrelated sources or cubes. This function allows analysts to perform analysis on data from multiple sources without integrating them into a single cube.

Aggregations and Calculations

• calculations and aggregations such as sum, average, count, minimum, maximum, and variance. Users perform these operations across one or more dimensions.

Streaming Databases

Introduction

Traditional Transactional Databases don’t scale well and may take hours to run complex SQLs with joins, aggregations, and transformations in analytical databases.

Streaming Databases serve results for such complex SQLs with sub-second latency, and provide fast continuous data transformation capabilities not possible in traditional databases.

Streaming Databases use SQL and familiar RDBMS abstractions (viz. tables, columns, rows, views, indexes), but have a completely different engine (a stream processor) and computing model (dataflows) inside.

Traditional Databases	Streaming Databases
store data in tables matching the structure of the inserts & updates and all the computation work happens on read queries.	ask for the queries upfront in the form of Materialized Views, incrementally update the results of these queries as input data arrives. So, Streaming databases move the computation work to the write side.

Origins

Early came about in the capital markets vertical, where value of fast computation over continuous data is very high e.g. StreamBase and KX System . These early generation of were more event processing frameworks than databases, and optimised for unique requirements of hedge funds and trading desks and not universality and accessibility.

While SQL-like control languages were implemented by the early ones (e.g. StreamBase - created with DDL statements like CREATE INPUT STREAM ), but the users had to be streaming systems experts.

SQL below doesn’t care if the data is static or actively updating. It has the info a streaming database needs to continually provide updated result sets as the soon as the data changes.

--- Sum revenue by product category

SELECT categories.name as category, SUM(line_items.amount) AS total_revenue
FROM purchases
JOIN line_items ON purchases.id = line_items.purchase_id
JOIN products ON products.id = line_items.product_id

JOIN categories ON product.category_id = categories.id
WHERE purchases.is_complete
GROUP BY 1;

ksqlDB and Flink allow users to define transformations in SQL but users still need to understand challenging streaming concepts to work around, like eventual consistency.

Recent focus in streaming databases is on expanding access to streaming computation by simplifying the control interface so that it can be operated by those familiar with traditional databases. Thus making application of streaming databases easier. 

Example Architectures

Streaming databases are often used “downstream” of primary transactional databases and message brokers, similar to how a Redis cache or a data warehouse might be used.

• A message broker reliably and continuously feeds streams of data into the database
• A Change Data Capture (CDC) service translates primary DB updates into structured data messages in to the message broker
• the SQL transformations are managed in dbt, as is in data warehouses
• user-facing applications and tools connect directly to the streaming database, with no need for caching and with more flexibility as compared to data warehouses

Are useful...

• to build realtime views with ANSI SQL to serve realtime analytics dashboards, APIs & apps
• to build notifications/alerting e.g. in fraud and risk models, or in building automated services that use event driven SQL primitives
• to build engaging experiences with customer data aggregations that should always be up-to-date e.g. personalisation, recommendations, dynamic pricing etc.

Not useful for solutions...

• that need columnar optimisation
• using window functions and non-deterministic SQL functions like RANK(), RANDOM() (While straightforward in traditional databases, running a these functions may result in continuous chaotic noise for streaming databases)
• with ad-hoc querying, as response times are compromised since the computation plan is not optimized for point-in-time results

Perform and scale well because...

• incremental updates ensure that DB update does not slow down as the dataset scales
• pre-computed query pattern as a persistent transformation ensure that reads are fast as no computation is required, it is just key-value lookups in memory, like Redis cache
• high frequency of concurrent reads from materialized views has minimal performance impact as complex queries with joins & aggregations are handled as persisted computation
• Aggregations are dramatically improved since

                    Resources requirement to handle persistent transformations.   

                                              ∝                           

                    Number of rows in the output (instead of the scale of the input)

May not perform and scale well...

• since SQL transformations are always running, joins over large datasets need a significant amount of memory to maintain intermediate state/s (Imagine how you would incrementally maintain a join between two datasets: You never know what new keys will appear on each side, so you have to keep the entirety of each dataset around in memory.)
• when a single change in inputs triggers change in output in many views, (or when many layers of views depend on each other) more CPU is required for each updat
• data updates trigger more work in DB requiring more CPU, than with data changes rarely
• high number of total unique keys slows down read queries in traditional databases. In streaming databases, high-cardinality increases initial “cold-start” time when a persistent SQL transformation is first created, and requires more memory on an ongoing basis

AI Opportunities in 2023 - Lecture by Dr. Andrew Ng

AI as a general purpose tech...

• is useful for lots of different applications e.g. electricity is good for a lot of things
• AI collection: Supervised learning and Generative AI (in focus today) + Unsupervised learning and Reinforcement learning
• Supervised learning: Good for labelling things e.g. 

                    e-mail >> spam or not, ship-route >> fuel consumed, Ad & user-info >> will click

• Workflow of Supervised learning apps: e.g. restaurant reviews classification

                            Collect dataset >> label data >> train a model >> deploy >> run

• Last decade was the decade of large scale supervised learning. Small AI models could be built on not very powerful computers, which had good performance for certain small amount of data but with even large amount of data the performance would flatten out. With large AI models, however, the performance scales better and better with large data
• This decade is adding to it the excitement of Generative AI
• When we train a very large AI model on a lot of data, we get a LLM like ChatGPT
• RLHF and other techniques tune AI output to be more helpful, honest and harmless
• And at the heart of Generative AI is (Supervised learning) repeated prediction of next sub-part patterns given the data it has seen
• The power of LLMs as a developer (not programmer) tool: 
• With prompt based AI - the workflow is:

                    Specify prompt >> Deploy to cloud (e.g. build restaurant review system in few days)

• Opportunities: massive value will be created with Supervised learning and Generative AI together, by identifying and executing concrete use cases
• Supervised learning will double in size and Generative AI will much more than double
• for new start-ups and for large enterprises / companies
• Lensa was an indefensible use case as it did not add value; AirBnB or Uber are defensible because these create value 
• The work ahead is to find the many diverse, value adding and defensible use cases

• Refer to the "Potential AI projects space curve"
• Advertising, and web search are the only large money making domains, with millions of users
• As we go to the right of the curve, some example projects of interest may be:
• Food inspection: cheese spread evenly on a pizza
• Wheat harvesting: how tall is the wheat crop, at what height should it be chopped off
• Materials grading, cloth grading...
• Clearly industries other than advertising and web-search have a very long tail of $5 mn projects but with a very high cost of customisation
• So, AI community needs to continue building better tools to help aggregate such use cases and make it easy for end users to do the customisations at affordable costs

Instead of needing to worry about pictures of pizza AI community will create tools to enable the IT department of the pizza factory to train an AI system on their own pizzas. Thus, to realise the value of $5 mn by leveraging some low/no code tools for AI

Referring to the AI Stack...

• H/W semi-conductor layer at bottom is very capital intensive and very concentrated
• Infrastructure layer above the semi-conductor later is also highly capital intensive and very concentrated
• Developer tools layer is hyper-competitive, and only a few will be winners
  All the above said layers can be successful only if the
• All the above said layers can be successful only if the application layer on top is even more successful e.g. Amorai - app for romantic relationships coaching
• Recipe for building startups, "don't rush to solutions", has been inverted and now we can just do that while still keeping it cost effective

              Ideas >> Validate >> Get CEO >> Prototype (early users) >> Pre-seed Growth (MVP) >> Seed, Growth & Scale

• Concrete ideas can be validated or falsified efficiently
• Even highly profitable projects but low on ethics will / should be killed
• AGI is still decades away
• Other areas of interest may be predicting next pandemic, climate change predictions...