Stuff I worked on

Technical Interview Notes

2016-07-12T13:34:00-04:00

Sorting

Divide and conquer: Splitting a problem into smaller problems similar to the initial until a point is reached where the problem is small enough that it can be solved on its own. Aftwerards, the solution is propogated back up to help solve the higher problem.
Inplace: Elements in the array are swapped when sorted instead of having a new sorted array returned.

Merge Sort

Divide and conquer sorting algorithm that sorts by splitting up an array into smaller chunks to be sorted. The chunks are split halfway until they are each of size 1. In this case, the chunks are already sorted. After dividing, pairs of sorted arrays are joined together, sorting them as they are merged until there is only one chunk left that is the final sorted array.

Best, average, and worst case time complexities are all O(nlog(n)). The complexity of the merging step is O(nlog(n)) since each merge involves iterating through both chunks in each pair, which grows with the length of the original array (n) and the number of merges decreases by a factor of 2 since half the number of chunks to merge remain after each process of merging.
The space complexity is O(n) since the sorted elements need to be stored in some intermediary when merging. This number increases based on the length of the initial array. As a result, the algorithm is also not inplace.

Quicksort

Divide and conquer algorithm that sorts very similarly to merge sort, but uses a pivot and wall to swap elements in place, avoiding having to create temporary arrays when merging. Quicksort works by partitioning the array into two partitions based on a pivot such that everything to the left of the pivot is less than it, and everything to the right of the pivot is greater than it. The new position of the pivot after the partitioning is kept track of by an incrementing wall/counter for the start of the array. The pivot itself is an arbitrary element in the array.

The partitioning works by iterating through all the elements of the array, swapping elements that are less than the pivot with the element immmediately to the left of the wall (which starts at index 0). Aftwerwards the wall inc incremented by 1, moving the wall to the right of the newly formed partition. After iterating through all elements, the pivot is swapped with the current element at the wall. At this point, the array is divided into 3 sections: the left partion of elements less than the pivot, the pivot itself, and the right partion of elements greater than (or equal to) the pivot.

The algorithm continues by applying this partitioning and sorting on the newly formed left and right partitions until the partions are of size 1, in which case, they are sorted. After having gone through all sub-partitions, the entire array has been sorted inplace.

The main action done in this algorithm behind the sorting is the swapping of elements, which has a constant space and time complexity since swapping just involves having a single temporary variable.
The wort case time complexity is $O(n^2)$ where the partitions are very imbalanced, causing either the left or right partion to always be of size 0. In this case, the division step creates a new partition whose length is just 1 less the previous which must still be iterated over to produce another partition. Both the iteration in each partition, and the number of partitions made increase with the length of the array, n. This scenario occurs in the case where either the smallest or largest elements in the array are chosen as the pivot for all partions since all other elements must be to the left of the partition if the max is chosen and to the right of the partition if the min is always chosen. To make sure the min and max aren't always selected, the median of the first handful of elements in the array is used as the pivot.
The average and best case time complexities are both O(nlog(n)). For best case, this occurs if the sub-partitions created are of equal size since the size of each sub partiion decreases by a factor of 2 each time. Always happenning to select the median of the partition can lead to this. The average case involves complex math, so see Khan Academy for a better explanation.
The space compelxity is constant since the array is sorted inplace and all the swaps just use 1 temporary variable.

Hash Tables

Hash tables (also known as dictionaries, maps, associative arrays) are abstract data types where each value is associated with a unique key. The elements themselves are actually stored in an array. The element index/offset at which these elements are stored is the result of a hash function for a given key. This hash function is responsible for converting a given data type to an integer representation of it for this offset.

Hash Function

The hash function itself should be implemented in such a way that the hash for a variable can be calculated quickly (near constant time), and each hash produced should be unique for every unique key. As a result, most of the time, hash functions involve some complicated math involving prime numbers, and coming up with a universal hash function that works for nearly all data types is impossible. (See python's implementation for string hashing as an example of how hashing works. I tend to use python's hashing functions when implementing my own hash tables.) If a hash function produces a hash whose value is at least the length of the array, the hash hash is modulo'd with the length of the array to produce an index that can be used on this array.

Collisions

Regardless of how well the hashing algorithm is, there is a possibility that two keys could produce the same hash, in which case, if an element already occupies the space for that hash, the element will still have to be stored somehow (it should not be dropped.)

In the event collisions occur, the hash function should be able to produce a uniform distribution of hashes, allowing for all elements to be accessed in an equal amount of time. A bad hash function produces a distribution of hashes such that a majority of the resulting hashes are the same or close to each other, resuliting in a distribution with peaks. If the hashes produced for a wide range of keys is the same, lookup is similar to lookup in a list.

Two ways to handle keys with duplicate hashes are through separate chaining and open addressing:

Separate chaining involves having each element in the hash table be a list (typically a linked list), and just appending an element at this hash to the linked list. This way, values for keys with duplicate hashes can still be stored at these hashes.
Open addressing involves placing values in the next available empty space in the array. If the space for a given hash is already occupied, the next available space is selected according to some probe sequence which returns the next hash to use for a given hash. Common ones include linear probing which just increments the hash by 1 until an empty space is found. Quadratic probing involves incrementing the hash by the square of the kth iteration into the probing function.

Both operations involve iterating over some sequence when collisions occur, effectively scaling up lookup linearly. In order to prevent collisions, some hash table implementations will dynamically resize the array to allow for more hashes to be stored. This assumes that the collisions are primarly a result of modulo-ing against the length of the array and not a result of the distribution of hashes formed by the funciton itself. I believe python's dict implementation uses a combination of open addressing and resizing the array if 2/3 of the array is occupied.

Operations

Lookup

This involves checking if an element for a given key exists in the hash table, which is essentially just running the key through the hash function. If chaining or open addressing is implemented, in order to retrive the proper element for a given key, the value of the key is also compared against against any subsequent value in the list or probe sequence, making the time complexity for lookup linear in worst case. This can be avaided though by resizing the array if a certain capacity threshold is reached to reduce the number of collisions.

Insertion/Updating

This involves inserting a value for a given key at an index in the array. If open addressing or chaining is implemented, and a collision occurs, the value is instead placed at the end of the list or the next avaialble spot found through probing.

Deletion

This varies depending on implementation. From a high level perspective, deleting the element could just mean setting the value at the hash to be NULL and decreasing a counter for the length of the array by 1. If separate chaining is implemented, deletion on the list implementation will take place for the given key and may involve iterating over the whole list. If open addressing is implemented, you will need to replace the deleted with a marker indicating the offset of the next element that should be checked as a result of probing.

Complexity

The main benefits behind hash tables are constant lookup, insertion, and deletion time. In the worst case scenario, when a bad hash funciton is used, the number of values for a given key can increase linearly, effectively making lookup time complexity that of whatever you implemented to ammend collision, but this can be countered by resizing the hash table. For average and best case scenarios, these operations are effectively done in constant time.

The cost of the hash table though is the amount of space needed. In order to support a large number of hashes, a large array will be needed to store all the elements. Objects in python are actually implemented as dictionaries in the underlying C code, and since everything in python is an object, this is one of the reasons for why python programs typically use much more memory than other languages.

Adding Custom Builtin Functions to the Python Interpreter

2016-06-14T14:03:00-04:00

When I have a few minutes of spare time, like if I’m on the bus heading to work or waiting for class to start, one of the things I like to do is explore the cpython source code. I want to learn how the Python interpreter works as a way to become a better Python programmer (or just a better programmer in general), and perhaps someday contribute to the development of Python in later releases.

Objective

One of the first things I did to get more familiar with the source is add my own custom builtin functions to the interpreter. I’m not talking about making a new function in python like def func(args). I’m also not talking about making an extension module in C that can be imported in python like from my_c_module import my_c_func. I’m talking about adding a new one of these functions: functions like sum(), abs(), and range() that are automatically built into the langauge and do not require manually importing anything to use. I figured this would be a good way to learn more about the source since I’d be learning how the core functions, which are part of the source, are implemented and used.

The function I will be adding for this example is a simple product function which will take the product of a list of numbers, similar to sum, but with multiplying instead of adding and a starting value of 1. Now, I know the idea of a product function has been proposed before and rejected due to low demand for the function, and there are many existing ways to impliment this function in python, but this is just for learning purposes.

Python Version

The version of python I am also working with is the first alpha version of Python 3.6.0 (more specifically, 3.6.0a1). The reason I am working with python 2 instead of 3 is because nearly all active development is in python 3, and the latest version of python 2 at the time (2.7.11) is only receiving bug fixes and backporting from python 3. There is a whole PEP dedicated to why Python 2.8 will never be a thing. You know it’s serious because the PEP number is 404. The reason I am also working with an alpha version of python 3 instead of the latest release at the time (3.5.1) is because I accidentally cloned the cpython repo from github without asserting first that the master branch contained the latest release version instead of development version.
¯\_(ツ)_/¯

How it works

Now, I kind of lied when I mentioned not making a C extension. It turns out that adding a builtin function is a lot like making a C extension for python. There are already lots of articles and documentation on the internet about making C extensions, especially this one, so I will not go into great detail about making one from scratch, just how it gets added and what changes I made to the source.

When you normally make a C extension, you add a PyMethodDef struct to an array of PyMethodDef structs which represent the functions you would like to expose in python space. These PyMethodDef structs is given as:

struct PyMethodDef {
    const char *ml_name; /* The name of the built-in function/method */
    PyCFunction ml_meth; /* The C function that implements it */
    int ml_flags; /* Combination of METH_xxx flags, which mostly describe the args expected by the C func */
    const char *ml_doc; /* The __doc__ attribute, or NULL */
};

Hopefully the comments in this code, taken straight from the source, are enough to explain the individual members of the struct.

Inside Python/bltinmodule.c, the array of PyMethodDef structs this gets added to is:

static PyMethodDef builtin_methods[] = {
    {"__build_class__", (PyCFunction)builtin___build_class__, METH_VARARGS | METH_KEYWORDS, build_class_doc},
    {"__import__", (PyCFunction)builtin___import__, METH_VARARGS | METH_KEYWORDS, import_doc},
    BUILTIN_ABS_METHODDEF
    BUILTIN_ALL_METHODDEF
    …
    {"vars",            builtin_vars,       METH_VARARGS, vars_doc},
    {NULL, NULL},
};

In the source, some of the array elements are given as literal structs while others like BUILTIN_ABS_METHODDEF are implemented as macros. You may notice that this array does not contain all builtin functions. The remaining functions like bytearray(), int(), or str() are actually constructors located in the _PyBuiltin_Initmethod also in Python/bltinmodule.c. (I don’t like using the word constructor when referencing these in python, but that’s another story.)

So, in order to add my custom builtin method, I just need to add a PyMethodDef to this array containing my function name, implementation, argument flags, and docstring.

Implementation

All changes implemented are in my copy of the source on Github.

I decided to try and isolate my additions as much as possible from the existing source so that it would be easier to reference later by making my changes stand out. The actual implementation of my builtin product function is pretty much an exact copy of the builtin sum function with a few minor changes.

Custom/custom.c

This file contains the implementation of my product function and the wrapper for it.

The implementation (builtin_prod_impl) does the exact same stuff as builtin_sum_impl, but with the default value changed from 0 to 1, the actual addition done by PyNumber_Add changed to PyNumber_Multiply for multiplication, and Fast Addition was removed. This Fast Addition was a way to speed up addition by storing the temporary sum in C space instead of Python space. The downside of this is that you always need to check for overflow since you’re working with data types that have a limited range of values. Now this can easily, and quickly, be done by comparing the sign bit of your result against the sign bit of your two numbers that you’re adding (which the interpreter does in this fast addition). However, for multiplication, I cannot find a quick way for checking if overflow occurred that does not involve dividing the product by one of the two numbers to get the other. Regardless, I ended up just using python’s builtin objects for multiplication since they handle overflow and large numbers already.

I have not profiled this, but I am curious to see if multiplying in C and dividing to check for overflow will still outperform PyNumber_Multiply.

The wrapper (builtin_prod) essentially unpacks the arguments into the iterable and starting value for the implementation.

Custom/custom.h

This file just contains the declarations for builtin_prod_impl and builtin_prod, the docstring, and the final macro to be included in the original builtin_methods array.

Python/bltinmodule.c

To actually my prod() function to the builtin functions, I just needed to add my macro to the array of builtin functions.

static PyMethodDef builtin_methods[] = {
    {"__build_class__", (PyCFunction)builtin___build_class__, METH_VARARGS | METH_KEYWORDS, build_class_doc},
    {"__import__", (PyCFunction)builtin___import__, METH_VARARGS | METH_KEYWORDS, import_doc},
    BUILTIN_ABS_METHODDEF
    BUILTIN_ALL_METHODDEF
    …
    {"vars",            builtin_vars,       METH_VARARGS, vars_doc},
#ifdef USE_CUSTOM_BUILTINS
    BUILTIN_PROD_METHODDEF
#endif
    {NULL, NULL},
};

The USE_CUSTOM_BUILTINS is just a flag I can pass to the compiler that says whether or not I want to include whatever custom builtin functions I had.

Makefile

The last thing to do was just adjust the Makefile such that it accepted this flag, and compiled the product function. Nothing big.

Usage

After re-building python from source, I am able to use prod() in both the shell and from a python script.

>>> prod(range(1, 6))
120
>>> prod([])  # Empty iterable, default starting value of 1
1
>>> prod(range(5))  # Zero included
0
>>> prod(range(-3, 4, 2))  # Negative numbers, positive result
9
>>> prod(range(-5, 4, 2))  # Negative numbers, negative result
-45
>>> prod((1, 2), 10)  # New starting value
20
>>> prod([2, 2**64]) == 2**65
True

Conclusions

Hopefully, this serves as a nice introduction into how new builtin functions are added. Feel free to send me an email if there’s anything blatantly wrong in this article.

Resources

Source

Fourier Transform Notes

2016-06-13T16:13:00-04:00

These are just notes I put on this website so that I will be able to remeber the content and be able to review it easly for later exams. This is not meant to go into full detail about the Fourier Transform, so stuff like derivations and proofs will not be included b/c deriving this stuff requires a lot more research than I'd like to do.

Fourier Transform

Fourier Transform

$$ F\{x(t)\} = X(\omega) = \int_{-\infty}^{\infty} x(t)e^{-j \omega t}dt $$

Inverse Fourier Transform

$$ F^{-1}\{X(\omega)\} = x(t) = \frac{1}{2\pi}\int_{-\infty}^{\infty} X(\omega)e^{j \omega t}d\omega $$

One of the main purposes of the Fourier Transform is to point out the dominant frequencies of a waveform. This can easily be seen with the FT of a regular sine or cosine wave.

The FT of a cosine wave is the dirac delta functions at $+ω_0$ and $-ω_0$. This makes sense since the only frequency in a regular cosine wave is $ω_0$. The reason for the peak at $-ω_0$ is because the FT of real waves are always symmetrical across the initial phase shift. In this case, φ = 0, so the FT of $cos(2πt)$ will be symmetrical across the y-axis.

For the example above, the FT is not an exact dirac function since I am only integrating over a finite set of data rather than an infinite set from t = -∞ to +∞. If I had more data points, the peaks would stand out more. Similarly, the reconstruction of x(t) from X(ω) is not exactly the same as the original x(t) because of the finite amount of data points I am integrating over in the frequency domain. For the continuous FT, the FT converts continuous data in the time domain to continous data in the frequency domain.

Discrete Time Fourier Transform

Discrete Time Fourier Transform of x[n]

$$ X_{s}(\Omega) = \sum_{n=-\infty}^{\infty} x[n]e^{-j \Omega n} $$

Inverse Fourier Transform of X(ω)

$$ x[n] = \frac{1}{2\pi} \int_{2\pi} X_s(\Omega) e^{j \Omega n} d\Omega $$

In the real world though, data is not necessarily continuous. In the example used to generate x(t), I just found x(t) at very small increments of t to replicate continuous data. Really, I just took discrete values of x(t) sampled every Ts seconds. The DTFT allows us to essentially take the Fourier Transform of discrete/sampled data, though the spectrum is different in the frequency domain than the continuous FT.

In the example above, I take the FT of a continuous cosine wave and the DTFT of the cosine wave sampled every Ts seconds. In this case, Ts = 0.05 s and my sampling frequency (fs) = 20 Hz. The FT of x(t) returns X(ω), which should be two dirac delta functions at positive and negative $ω_0$. The DTFT of x[n], however, returns Xs(Ω). This spectrum is different from that of the FT in that:

The DTFT is in terms of Ω while the FT is in terms of ω, where $X_s(Ω)$ = $X_s(ωTs)$. (A similar example of this notation is how x[n] = x(nTs).) Though both have the same units (rad/s), the scale of the DTFT is smaller than that of the FT by a factor of the sampling frequency (fs = 1/Ts).
The DTFT repeats every 2π. This is because the DTFT is the sum of various FTs at frequencies that are 2π apart from each other.
The DTFT transforms discrete data from the time domain to continuous data in the frequency domain while the FT transforms continuous data in the time domain to continuous data in the frequency domain.

Like the IFT, the original discrete signal can also be reconstructed from its DTFT. Theoretically, the continuous signal could be constructed from the discrete signal, though this would require an infinitely large sampling frequency to replicate an infinitesimally small dt.

Resources

Fourier Transforms Table

Source Code

Matlab
- Cosine Fourier Transform
- Cosine Discrete Time Fourier Transform

Test Github Post

2016-06-12T15:37:00-04:00

This is a placeholder for Github Projects category.