Showing posts with label Technical. Show all posts
Showing posts with label Technical. Show all posts

Friday, November 03, 2023

Managing Technical Debt

Introduction

The gap between the current state of a software system e.g. inefficient code, design flaws, missing documentation etc. is usually called technical debt. It represents the future cost that will be incurred when addressing such shortcomings e.g. as fixing bugs, refactoring code, improving documentation, making the system more secure and scalable.

Technical debt, incurred intentionally or unintentionally is not necessarily a bad thing in all cases. In some situations, technical debt results from strategic decisions made to meet urgent business needs or to gain a short term but competitive advantage. However, it must be managed and paid down over time to prevent any overwhelming hinderances to future development. 

Technical Debt Management Framework

To effectively manage technical debt, it must be recognised, documented, and tracked as part of the software development process. This helps make informed decisions about when and how to address the debt by balancing short-term goals with long-term sustainability and quality.

The industry recognises and categorises a number of forms of technical debt. However, the following are of most interest for an investment banking enterprise:
Difficulty column gives an indication of effort that may be invested in managing and repaying a given category of the debt. As an example, the commonly identified 'Architecture and Design Debt'  is quite difficult to identify and measure in hindsight, because the decisions are soundly supported by business needs. Usually 'Architecture and Design Debt' is also closely related to other debts e.g. 'Integration Debt' and thus very difficult to even monitor and track. This debt is also very difficult to repay since architecture and design are rarely revisited without motivation and certainly not only with just the intent to repay by remediating and refactoring code.

The framework considers 'Architecture and Design Debt' as an umbrella encompassing 'Code Debt', 'Performance Debt' and others as remediation sub-categories (add a column) with the following considerations:
  • Performance Debt: Monitor performance using APM tools, identify performance bottlenecks | Set performance benchmarks, track metrics | Continuously monitor performance, allocate resources to address issues | Variability of workloads, difficulty in isolating performance causes | 
  • Code Debt: Evaluate code quality using metrics such as cyclomatic complexity and code duplication | Maintain code documentation | Conduct code reviews, implement automated code quality checks | Complexity of large codebases, rapidly evolving code
One may want to add 'Skills Debt' to the framework. It is medium difficulty, can be more scientifically measured & remediated and helps to repay other types of technical debts by a well managed and self motivated team.

The framework as well identifies challenges to expect and lists some generic steps that could be taken to alleviate the issues while actively managing and repaying technical debt.

Saturday, March 21, 2015

Sybase 12.5 to Sybase 15.5 migration - differences that I learnt about

1. Login triggers were introduced in ASE 15.0. A regular ASE stored procedure can automatically executed in background on successful login by any user.

2. Fast bcp is allowed for indexed tables in ASE 15.0.2 and above. bcp works in one of the two modes

  • Slow bcp - logs all the row inserts made, is slower and is used for tables that have one or more index
  • Fast bcp - only page allocations are logged, used for tables without indexes, used when fastest speed possible is required, can be used for tables with non-clustered indexes
3. sp_displaylogin displays when and why a login was locked & also when you last logged in. 

4. Semantic partitions/smart partitioning: ASE 15 makes large databases easy to manage and more efficient by allowing you to divide tables into smaller partitions which can be individually managed. You can run maintenance tasks on selected partitions to avoid slowing overall performance, and queries run faster because ASE 15's smart query optimizer bypasses partitions that don't contain relevant data 

5. With large data sets, filing through a mountain of results data can be difficult. ASE 15's bi-directional scrollable cursors make it convenient to work with large result sets because your application can easily move backward and forward through a result set, one row at a time. This especially helps with Web applications that need to process large result sets but present the user with subsets of those results 

6. Computed columns: Often applications repeat the same calculation over and over for the same report or query. ASE 15 supports both virtual and materialized columns based on server calculations. Columns can be the computed result of other data in the table, saving that result for future repeated queries 

7. Functional indexes: When applications need to search tables based on the result of a function, performance can suffer. Functional indexes allow the server to build indexes on a table based on the result of a function. When repeated searches use that function, the results do not need to be computed from scratch 

8. Plan viewer in the form of a GUI: Plans for solving complicated queries can become very complex and make troubleshooting performance issues difficult. To make debugging queries simpler, ASE 15 provides a graphical query plan viewer that lets you visualize the query solution selected by ASE's optimizer.

9. In ASE 15.0, Update statistics and sp_recompile are not necessary after index rebuild

10. ASE 15 allows to assign two billion logical devices to a single server, with each device up to 4 Tb in size. It supports over 32,767 databases, and the maximum size limit for an individual database is 32 terabytes, extending the maximum storage per ASE server to over 1 million terabytes!

11. As of release 12.5.1, all changes to data cache are dynamic

12. ASE 15.0 and later versions no longer use vdevno. i.e. the disk init syntax doesn't need to mention the vdevno parameter. 

13. Disk init syntax in 12.5 expects size parameter in K, M, and G only. From 15.0 and onwards, T (Terabyte) can be specified. 
Also, pre 15.0; the maximum size of a device was 32GB 

14. The configuration parameter ?default database size? was static in ASE 12. In ASE 12.5, it was made dynamic. 
For ASE 15.0, the below table is specified by Sybase. 
Logical Page Size 2K 4K 8K 16K 
Initial default database size 3MB 4 MB 8 MB 16 MB 
All system tables, initially 1.2 MB 2.4 MB 4.7 MB 9.4 MB

15. The auto database extension was introduced in 12.5.1 and supported later versions.

16. The Dump/Load Database and Dump/Load Tran syntax differ from version 12.5.0.3 and 12.5.2(and hence later versions) (See sybooks for more information. The compression levels 1-9 were introduced.) 

17. ASE 12.5.0.3 and earlier versions used to allow only one tempdb in the server. But all the later versions allow creation of multiple temporary databases. 

18. Before 15.0, after changing a database option we need to use that database and do checkpoint on it. But ASE15.0 doesn't need this. 

19. Restricting proxy authorization is available in 12.5.2 and later releases only. 

20. From version 12.5.2 and onwards, cache creation is made dynamic (sp_cacheconfig [cachename [,"cache_size [P|K|M|G]?] It was static earlier. 

21. Till 12.5.2, backing up a database with password was not possible. ASE 12.5.2 and later allow dump database with passwd. 

22. Cross platform dumps and loads were introduced in ASE 12.5.3 

23. MDA tables (Monitoring and Diagnostic Tables) are available in 12.5.3 and later releases. 

24. Row Level Locking: In ASE 15.0 all the system tables were converted into datarows format.

25. Group By without Order By

Thursday, January 08, 2015

Passing an array by reference

I am blessed to have colleagues to debate with, interesting ones. Today's was about passing an array as a reference. Usually one ends-up passing the array as pointer and the size in the second parameter.

Given below is the code where one is able to pass to a function an entire array of fixed size:


#include <iostream>

int func10(int (&tenIntArray)[10])
{
for (int i = 0; i < 10; i++)
{
if (i == 0) continue;
tenIntArray[i] = tenIntArray[i - 1] + 1;
}
return 0;
}

// Only to show how it is typedef'd
typedef int FiveIntArray[5];

int func05(FiveIntArray &fiveIntArray)
{
return 0;
}

int main(int argc, char* argv[])
{
int tenIntArr[10] = { 1 };
FiveIntArray fiveIntArr = { 1 };

func10(tenIntArr); // we can't pass fiveIntArr here

for (int i = 0; i < 10; i++)
{
std::cout << tenIntArr[i] << std::endl;
}

func05(fiveIntArr); // a tenIntArr is not acceptable here

return 0;
}


It is easier to understand if one has typedef it.

The general rule with the use of typedef is to write out a declaration as if you were declaring a variable of the type you want. Where the declaration would have given a variable name of the particular type, prefixing the whole thing with typedef gives a new type-name instead. The new type-name can then be used to declare new variables.

Tuesday, January 06, 2015

Monte Carlo simulations to determine the value of PI

Consider a circle with radius R. The area of the circle is PI x R^2. The area of the square that just embeds our circle (as in image below) is (2R)^2.


One takes a print of the image, sticks it on a cardboard and throws a dart at it. The chance the dart will fall within the circle is the ratio of area of the circle to the area of the square.

Chance of dart landing within the circle =  Area of circle 
                                                                    Area of square

                                                                 =  PI x R^2 
                                                                       (2R)^2

                                                                 =  PI x R^2 
                                                                      4 x R^2

4 x (Chance of dart landing within the circle) = PI

Whether the dart is within the circle or beyond the circle is simply determined by the distance at which it is from the centre of the figure (as illustrated).



To determine the chance that a dart thrown at the figure will land within the circle, one throws the dart several times and notes the distance of its landing from the origin. 

The ratio  number of times the dart lands at a distance less than equal to the radius of the circle 
                                                              total number of throws

gives us the probability (i.e. chance) of the dart landing within the circle in total number of throws.

There is no bias expected. Hence, the probability of the dart falling within the quarter circle shown in the red square above is the same as that of the dart falling within the entire circle.

Programatically one now needs to randomly select (x, y) to represent a point at which the dart lands, determine if it is at a distance greater than R or not. If it was a unit circle (i.e. R = 1) the comparisons are a little easier since there is no need to take a root of (x^2 + y^2). 

Below is a program that gives us the probability of selecting a point (x, y) such that it falls in the quarter circle:


    #include <iostream>
    #include <ctime>
    
    void usage()
    {
     std::cout << "Usage: find_pi <number of tries>" << std::endl;
     return;
    }
    
    int main(int argc, char* argv[])
    {
     if (argc != 2)
     {
            usage();
            return -1;
     }
    
     int numTries = atoi(argv[1]);
    
     int total = 0;
     int inCircle = 0;
    
     srand(static_cast<unsigned>(time(0)));
    
     for (int total = 0; total < numTries; total++)
     {
            double x = static_cast<double>(rand()) / RAND_MAX;
            double y = static_cast<double>(rand()) / RAND_MAX;
            double z = x*x + y*y;
            
            if ((z - 1.0) <= DBL_EPSILON)
         {
     ++inCircle;
            }
     }
    
     std::cout << (double)inCircle * 4.0 / numTries << std::endl;
    
return 0;
    }

Monday, January 05, 2015

Constant pointer and pointer to a constant, yet another attempt

Master: Appears that 'pointer to a constant' and 'constant pointer' are not understood by so many C++ programmers.

Grasshopper: Ain't that simple, one could not change the value when it is a 'pointer to a constant' while the pointer could not be changed for a 'constant pointer'.

Master: That is what all of them recite like a nursery rhyme, rarely does one tell me what is what.

Grasshopper: Hmm....

Master: Hmm...

Grasshopper: Hmm?

Master: Right, amongst the many attempts around the world the below image is my effort. I really hope this will close the gap that I just noticed.





Grasshopper: I do not think it could be further simplified, I won't ever forget it. Thank you very much.

Master: <Smiles and bows> We could now talk of the differences in them.

Grasshopper: Which become so obvious now...

Sunday, January 04, 2015

Dynamically assign memory to 2D array of integers

Understanding memory management is critical for C++ programmers. One of my colleagues asks people to write code to assign memory to 2D array of integers. The below program shows an easy way of doing that.


#include <iostream>

typedef int* IntPtr;

int main()
{
const int numCols = 5;
const int numRows = 20;

// assign memory for rows
IntPtr* intPtrArr = new IntPtr[numRows];

// now assign memory for columns in each row
for (int i = 0; i < numRows; i++)
{
intPtrArr[i] = new int[numCols];
}

// now assign values to each element
for (int i = 0; i < numRows; i++)
{
for (int j = 0; j < numCols; j++)
{
intPtrArr[i][j] = (i + 1)*(j + 1);
}
}

// now print whatever is in the array
for (int i = 0; i < numRows; i++)
{
for (int j = 0; j < numCols; j++)
{
std::cout << intPtrArr[i][j] << " ";
}
std::cout << std::endl;
}

return 0;
}