Neuromorphic Computing.

Avant-Garde-Technologies spends a large amount of money on the development of neuromorphic computer processors.   This technology will become vastly important in the design in computers that are use for AI and ML. Neuromorphic Computing refers to the designing of computers that are based on the systems found in the human brain and the nervous system. 

Driven by the vast potential and ability of the human brain, neuromorphic computing devises computers that can work as efficiently as the human brain without acquiring large room for the placement of software. 

Inspired by the human brain and the functioning of the nervous system, Neuromorphic Computing was a concept introduced in the 1980s. Yet this concept has taken the front seat in recent times as Artificial Intelligence has led scientists to advance Neuromorphic Computing to excel in the field of technology. 

Features of Neuromorphic Computing 

1. Rapid Response System

Neuromorphic computers are specifically known for their rapid response system because their processing is highly rapid. As compared to traditional computers, neuromorphic computers are built to work like a human brain and so their rapid response system is a major highlight. 

2. Low Consumption of Power

Owing to the concept of Spiking Neural Networks (SNN), neuromorphic machines work when electric spikes or signals are passed through the artificial neurons. These artificial neurons work only when electric spikes are passed through them thus consuming low energy. 

3. Higher Adaptability

Modern computers have a knack for adaptability and so do neuromorphic computers. With higher adaptability, neuromorphic computers work well according to the evolving demands of technology. With changing times, neuromorphic computers adapt themselves and change from time to time resulting in efficient working. 

4. Fast-paced Learning 

Machines working on the principle of neuromorphic computing are highly fast-paced when it comes to learning. Establishing algorithms based on interpretation of data and formulating algorithms as and when new data is fed into such computers, neuromorphic computing enables machines to learn rapidly. 

5. Mobile Architecture

One of the most striking features of neuromorphic computing is its mobile architecture. Unlike traditional computers that used to consume vast space for working, neuromorphic computers are mobile and handy. They do not require much space and are highly efficient in terms of space occupancy. 

Neuromorphic Computing and Artificial Intelligence 

Artificial Intelligence technology intends to impart human abilities in computers to make them work like humans. On the other hand, neuromorphic computing attempts to engineer computers that work like the human brain does. Comprising millions of artificial neurons that pass on electric signals to one another, neuromorphic computing has been a revolutionary concept in the realm of Artificial Intelligence. 

By inducing the technology of information processing, neuromorphic computers have become the leaders of AI that, as many say, have resulted in the 3rd wave. The third generation of AI has led scientists to draw parallels with the human brain and its abilities like the interpretation of data and adaptation. 

With the help of one of the techniques of AI, (machine learning), neuromorphic computing has advanced the process of information processing and enabled computers to work with better and bigger technology.  While the traditional computer software could only support Artificial Intelligence to a limited extent, modern age computers have come up to be faster, better, and brighter in terms of use and ability. 

Thanks to AI, neuromorphic computing has reinvented its place in the field of technology and is pushing the limits of AI further. Intertwined with each other, neuromorphic computing and AI have a long way to go as both attempt to mimic human abilities and imitate them in computer software. 

Examples of AI

Examples of AI are incorporated into a variety of different types of technology illustrates it enormous benefits.

• Automation. When paired with AI technologies, automation tools can expand the volume and types of tasks performed. An example is robotic process automation (RPA), a type of software that automates repetitive, rules-based data processing tasks traditionally done by humans. When combined with machine learning and emerging AI tools, RPA can automate bigger portions of enterprise jobs, enabling RPA's tactical bots to pass along intelligence from AI and respond to process changes.
• Machine learning. This is the science of getting a computer to act without programming. Deep learning is a subset of machine learning that, in very simple terms, can be thought of as the automation of predictive analytics. There are three types of machine learning algorithms:
  • Supervised learning. Data sets are labeled so that patterns can be detected and used to label new data sets.
  • Unsupervised learning. Data sets aren't labeled and are sorted according to similarities or differences.
  • Reinforcement learning. Data sets aren't labeled but, after performing an action or several actions, the AI system is given feedback.
• Machine vision. This technology gives a machine the ability to see. Machine vision captures and analyzes visual information using a camera, analog-to-digital conversion and digital signal processing. It is often compared to human eyesight, but machine vision isn't bound by biology and can be programmed to see through walls, for example. It is used in a range of applications from signature identification to medical image analysis. Computer vision, which is focused on machine-based image processing, is often conflated with machine vision.
• Natural language processing (NLP). This is the processing of human language by a computer program. One of the older and best-known examples of NLP is spam detection, which looks at the subject line and text of an email and decides if it's junk. Current approaches to NLP are based on machine learning. NLP tasks include text translation, sentiment analysis and speech recognition.
• Robotics. This field of engineering focuses on the design and manufacturing of robots. Robots are often used to perform tasks that are difficult for humans to perform or perform consistently. For example, robots are used in assembly lines for car production or by NASA to move large objects in space. Researchers are also using machine learning to build robots that can interact in social settings.
• Self-driving cars. Autonomous vehicles use a combination of computer vision, image recognition and deep learning to build automated skill at piloting a vehicle while staying in a given lane and avoiding unexpected obstructions, such as pedestrians.

There are many different types of neural networks used for AI.  The  problem with drawing them as node maps is that it doesn’t really show how they’re used. For example, variational autoencoders (VAE) may look just like autoencoders (AE), but the training process is actually quite different. The use-cases for trained networks differ even more, because VAEs are generators, where you insert noise to get a new sample. AEs, simply map whatever they get as input to the closest training sample they “remember”. I should add that this overview is in no way clarifying how each of the different node types work internally (but that’s a topic for another day).

It should be noted that while most of the abbreviations used are generally accepted, not all of them are. RNNs sometimes refer to recursive neural networks, but most of the time they refer to recurrent neural networks. That’s not the end of it though, in many places you’ll find RNN used as placeholder for any recurrent architecture, including LSTMs, GRUs and even the bidirectional variants. AEs suffer from a similar problem from time to time, where VAEs and DAEs and the like are called simply AEs. Many abbreviations also vary in the amount of “N”s to add at the end, because you could call it a convolutional neural network but also simply a convolutional network (resulting in CNN or CN).

Composing a complete list is practically impossible, as new architectures are invented all the time. While the list of neural networks may provide you with some insights into the world of AI, but this  list is not  comprehensive. 

For each of the architectures depicted in the picture, brief description is provided..

Radial basis function (RBF) networks are FFNNs with radial basis functions as activation functions. There’s nothing more to it. Doesn’t mean they don’t have their uses, but most FFNNs with other activation functions don’t get their own name. This mostly has to do with inventing them at the right time.

Long / short term memory (LSTM) networks try to combat the vanishing / exploding gradient problem by introducing gates and an explicitly defined memory cell. These are inspired mostly by circuitry, not so much biology. Each neuron has a memory cell and three gates: input, output and forget. The function of these gates is to safeguard the information by stopping or allowing the flow of it. The input gate determines how much of the information from the previous layer gets stored in the cell. The output layer takes the job on the other end and determines how much of the next layer gets to know about the state of this cell. The forget gate seems like an odd inclusion at first but sometimes it’s good to forget: if it’s learning a book and a new chapter begins, it may be necessary for the network to forget some characters from the previous chapter. LSTMs have been shown to be able to learn complex sequences, such as writing like Shakespeare or composing primitive music. Note that each of these gates has a weight to a cell in the previous neuron, so they typically require more resources to run.

Autoencoders (AE) are somewhat similar to FFNNs as AEs are more like a different use of FFNNs than a fundamentally different architecture. The basic idea behind autoencoders is to encode information (as in compress, not encrypt) automatically, hence the name. The entire network always resembles an hourglass like shape, with smaller hidden layers than the input and output layers. AEs are also always symmetrical around the middle layer(s) (one or two depending on an even or odd amount of layers). The smallest layer(s) is|are almost always in the middle, the place where the information is most compressed (the chokepoint of the network). Everything up to the middle is called the encoding part, everything after the middle the decoding and the middle (surprise) the code. One can train them using backpropagation by feeding input and setting the error to be the difference between the input and what came out. AEs can be built symmetrically when it comes to weights as well, so the encoding weights are the same as the decoding weights.

Denoising autoencoders (DAE) are AEs where we don’t feed just the input data, but we feed the input data with noise (like making an image more grainy). We compute the error the same way though, so the output of the network is compared to the original input without noise. This encourages the network not to learn details but broader features, as learning smaller features often turns out to be “wrong” due to it constantly changing with noise.

Sparse autoencoders (SAE) are in a way the opposite of AEs. Instead of teaching a network to represent a bunch of information in less “space” or nodes, we try to encode information in more space. So instead of the network converging in the middle and then expanding back to the input size, we blow up the middle. These types of networks can be used to extract many small features from a dataset. If one were to train a SAE the same way as an AE, you would in almost all cases end up with a pretty useless identity network (as in what comes in is what comes out, without any transformation or decomposition). To prevent this, instead of feeding back the input, we feed back the input plus a sparsity driver. This sparsity driver can take the form of a threshold filter, where only a certain error is passed back and trained, the other error will be “irrelevant” for that pass and set to zero. In a way this resembles spiking neural networks, where not all neurons fire all the time (and points are scored for biological plausibility).

A Hopfield network (HN) is a network where every neuron is connected to every other neuron; it is a completely entangled plate of spaghetti as even all the nodes function as everything. Each node is input before training, then hidden during training and output afterwards. The networks are trained by setting the value of the neurons to the desired pattern after which the weights can be computed. The weights do not change after this. Once trained for one or more patterns, the network will always converge to one of the learned patterns because the network is only stable in those states. Note that it does not always conform to the desired state (it’s not a magic black box sadly). It stabilises in part due to the total “energy” or “temperature” of the network being reduced incrementally during training. Each neuron has an activation threshold which scales to this temperature, which if surpassed by summing the input causes the neuron to take the form of one of two states (usually -1 or 1, sometimes 0 or 1). Updating the network can be done synchronously or more commonly one by one. If updated one by one, a fair random sequence is created to organise which cells update in what order (fair random being all options (n) occurring exactly once every n items). This is so you can tell when the network is stable (done converging), once every cell has been updated and none of them changed, the network is stable (annealed). These networks are often called associative memory because the converge to the most similar state as the input; if humans see half a table we can image the other half, this network will converge to a table if presented with half noise and half a table.

Restricted Boltzmann machines (RBM) are remarkably similar to BMs (surprise) and therefore also similar to HNs. The biggest difference between BMs and RBMs is that RBMs are a better usable because they are more restricted. They don’t trigger-happily connect every neuron to every other neuron but only connect every different group of neurons to every other group, so no input neurons are directly connected to other input neurons and no hidden to hidden connections are made either. RBMs can be trained like FFNNs with a twist: instead of passing data forward and then back-propagating, you forward pass the data and then backward pass the data (back to the first layer). After that you train with forward-and-back-propagation.

Convolutional neural networks (CNN or deep convolutional neural networks, DCNN) are quite different from most other networks. They are primarily used for image processing but can also be used for other types of input such as as audio. A typical use case for CNNs is where you feed the network images and the network classifies the data, e.g. it outputs “cat” if you give it a cat picture and “dog” when you give it a dog picture. CNNs tend to start with an input “scanner” which is not intended to parse all the training data at once. For example, to input an image of 200 x 200 pixels, you wouldn’t want a layer with 40 000 nodes. Rather, you create a scanning input layer of say 20 x 20 which you feed the first 20 x 20 pixels of the image (usually starting in the upper left corner). Once you passed that input (and possibly use it for training) you feed it the next 20 x 20 pixels: you move the scanner one pixel to the right. Note that one wouldn’t move the input 20 pixels (or whatever scanner width) over, you’re not dissecting the image into blocks of 20 x 20, but rather you’re crawling over it. This input data is then fed through convolutional layers instead of normal layers, where not all nodes are connected to all nodes. Each node only concerns itself with close neighbouring cells (how close depends on the implementation, but usually not more than a few). These convolutional layers also tend to shrink as they become deeper, mostly by easily divisible factors of the input (so 20 would probably go to a layer of 10 followed by a layer of 5). Powers of two are very commonly used here, as they can be divided cleanly and completely by definition: 32, 16, 8, 4, 2, 1. Besides these convolutional layers, they also often feature pooling layers. Pooling is a way to filter out details: a commonly found pooling technique is max pooling, where we take say 2 x 2 pixels and pass on the pixel with the most amount of red. To apply CNNs for audio, you basically feed the input audio waves and inch over the length of the clip, segment by segment. Real world implementations of CNNs often glue an FFNN to the end to further process the data, which allows for highly non-linear abstractions. These networks are called DCNNs but the names and abbreviations between these two are often used interchangeably.

Deep convolutional inverse graphics networks (DCIGN) have a somewhat misleading name, as they are actually VAEs but with CNNs and DNNs for the respective encoders and decoders. These networks attempt to model “features” in the encoding as probabilities, so that it can learn to produce a picture with a cat and a dog together, having only ever seen one of the two in separate pictures. Similarly, you could feed it a picture of a cat with your neighbours’ annoying dog on it, and ask it to remove the dog, without ever having done such an operation. Demo’s have shown that these networks can also learn to model complex transformations on images, such as changing the source of light or the rotation of a 3D object. These networks tend to be trained with back-propagation.

Liquid state machines (LSM) are similar soups, looking a lot like ESNs. The real difference is that LSMs are a type of spiking neural networks: sigmoid activations are replaced with threshold functions and each neuron is also an accumulating memory cell. So when updating a neuron, the value is not set to the sum of the neighbours, but rather added to itself. Once the threshold is reached, it releases its’ energy to other neurons. This creates a spiking like pattern, where nothing happens for a while until a threshold is suddenly reached.

Echo state networks (ESN) are yet another different type of (recurrent) network. This one sets itself apart from others by having random connections between the neurons (i.e. not organised into neat sets of layers), and they are trained differently. Instead of feeding input and back-propagating the error, we feed the input, forward it and update the neurons for a while, and observe the output over time. The input and the output layers have a slightly unconventional role as the input layer is used to prime the network and the output layer acts as an observer of the activation patterns that unfold over time. During training, only the connections between the observer and the (soup of) hidden units are changed.

Neural Turing machines (NTM) can be understood as an abstraction of LSTMs and an attempt to un-black-box neural networks (and give us some insight in what is going on in there). Instead of coding a memory cell directly into a neuron, the memory is separated. It’s an attempt to combine the efficiency and permanency of regular digital storage and the efficiency and expressive power of neural networks. The idea is to have a content-addressable memory bank and a neural network that can read and write from it. The “Turing” in Neural Turing Machines comes from them being Turing complete: the ability to read and write and change state based on what it reads means it can represent anything a Universal Turing Machine can represent.

Capsule Networks (CapsNet) are biology inspired alternatives to pooling, where neurons are connected with multiple weights (a vector) instead of just one weight (a scalar). This allows neurons to transfer more information than simply which feature was detected, such as where a feature is in the picture or what colour and orientation it has. The learning process involves a local form of Hebbian learning that values correct predictions of output in the next layer.

Attention networks (AN) can be considered a class of networks, which includes the Transformer architecture. They use an attention mechanism to combat information decay by separately storing previous network states and switching attention between the states. The hidden states of each iteration in the encoding layers are stored in memory cells. The decoding layers are connected to the encoding layers, but it also receives data from the memory cells filtered by an attention context. This filtering step adds context for the decoding layers stressing the importance of particular features. The attention network producing this context is trained using the error signal from the output of decoding layer. Moreover, the attention context can be visualized giving valuable insight into which input features correspond with what output features.

IT integration isn’t the same as continuous integration (CI), which is a developer practice where working copies of code are merged into a shared central repository multiple times a day. The goal of CI is to automate build and verifications so problems can be detected earlier—leading to faster development.  IT systems have sprawled out over the organization away from one another.  Since each vendor’s solutions do communicate between one another, the entire IT stack needs a way to organize this technology "spaghetti" and allow it to act on business logic.

The hub-and-spoke model, where the connections between apps and services are handled by a central broker knows as the hub. The spokes that connect the hub to the apps and services can be individually maintained. This allows the apps themselves to be more focused, with all arts of integrations handled via the hub and spokes. The main downside to this approach is the centralization of the hub. It becomes a single point of failure for the system and for your infrastructure communications. All integrations in the EAI hub-and-spoke model is designed to  be dependent on the hub to function.

The centralization of the hub is the major weakness, since it becomes a single point of failure for the system and for the infrastructure communications. All integrations in the EAI hub-and-spoke model is designed to be dependent on the hub to function.

Robot Process Automation (RPA)

Robotic process automation (RPA) is a software technology that makes it easy to build, deploy, and manage software robots that emulate humans actions interacting with digital systems and software. RPA is used to automate various supply chain processes, including data entry, predictive maintenance and after-sales service support.  Furthermore, RPA is used across industries to automate high volume, rote tasks. Telecommunications companies use RPA to configure new services and the associated billing systems for new accounts.

A major purpose for using RPA is to save time and reduce operating expenses.  For example, RPA is used Finance & Accounting Invoice Processing, Human Resources Hiring, Retail Inventory Management, Payroll and Customer Support.

When RPA is used with AI technologies (aka Intelligent Automation), it enables businesses to operate by seamlessly integrating technology, work processes, and people.  For example, RPA+AI will enable (1) Automation for any business process from start to finish, (2) Connect front- and back-office processes, (3) Organize and process complex data, (4) Eliminate errors and exceptions, (5) Strengthen operational security, (6) Ensure compliance, (7) Enhance customer experience, and achieve a very high ROI.

RPA bots create step-change in employee productivity by accelerating workflows and enabling more work to get done by executing processes independently. In document-intensive industries like financial services, insurance and in the public sector, RPA bots can handle form filling and claims processing all hands-free.

Automating with RPA is enabling industries such as finance, healthcare and life sciences to leverage the reliability of bots to achieve strict compliance standards. Robotic Process Automation in accounting is enabling new levels of speed and precision in order-to-cash and procure-to-pay processes.

Artificial intelligence (AI) is combined with RPA to create Intelligent Automation.  The automation extends by an order of magnitude and is able to draw on the 80% of enterprise data that’s unstructured.  For example, in procure-to-pay, it provides for automate invoice processing of non-standard vendor invoices. In insurance, automate extracting claims data and detecting potential fraud. In HR, automate request intake by understanding the employee’s intent.

Big Data 

Big Data contains greater variety of data, arriving in increasing volumes and with more velocity.   Big data is larger, more complex data sets, especially from new data sources.  These data sets are so voluminous that traditional data processing software just can't manage them.   Big data is classified as Structured Data, Unstructured Data and Semi-Structured Data.  There are also four dimensions of Big Data  that include volume, variety, velocity and veracity.   Big data i describes large, hard-to-manage volumes of data.   This includes both structured and unstructured data.   The amount of data accumulated continues to increase at an exponential rate.    It’s not just the type or amount of data that’s important factor.  What organizations do with the data that matters even more.. Big data is analyzed for insights that improve decisions and give confidence for strategic business planning.

Data scientists analyzes and looks for insights in data. Data engineers build pipelines focused on DataOps. Data officers ensure data is reliable and managed responsibly. Synergy among roles drives analytics success.

Big data projects demand intense resources for data processing and storage. Working together, big data technologies and cloud computing provide a cost-effective way to handle all types of data.  This leads to  a winning combination of agility and elasticity.

The data lake offers the ability to query smaller, more relevant and more flexible data sets. As a result, query times can drop to a fraction of what they would have been in a data mart, data warehouse or relational database.

The data lake emphasizes the flexibility and availability of data. It can provide users and downstream applications with schema-free data.  Data resembles its “natural” or raw format regardless of origin. Data lake applications do not support partial or incremental loading. This is the primary basic way the data lake differs from the data warehouse. Data cannot loaded or portions reloaded into a data lake.

Business intelligence covers the processes and methods of collecting, storing, and analyzing data from business operations or activities to optimize performance.  All of these things come together to create a comprehensive view of a business to help make better, actionable decisions. Today business intelligence has evolved to include more processes and activities to help improve performance.

• Data mining: Using databases, statistics and machine learning to uncover trends in large datasets.
• Reporting: Sharing data analysis to stakeholders so they can draw conclusions and make decisions.
• Performance metrics and benchmarking: Comparing current performance data to historical data to track performance against goals, typically using customized dashboards.
• Descriptive analytics: Using preliminary data analysis to find out what happened.
• Querying: Asking the data specific questions, BI pulling the answers from the datasets.
• Statistical analysis: Taking the results from descriptive analytics and further exploring the data using statistics such as how this trend happened and why.
• Data visualization: Turning data analysis into visual representations such as charts, graphs, and histograms to more easily consume data.
• Visual analysis: Exploring data through visual storytelling to communicate insights on the fly and stay in the flow of analysis.
• Data preparation: Compiling multiple data sources, identifying the dimensions and measurements, preparing it for data analysis.

There are a number of BI tools in the marketplace, which aid business users in analyzing performance metrics and extracting insights in real time. These tools focus on self-service capabilities and enabling decision-makers to recognize performance gaps, market trends, or new revenue opportunities more quickly. BI applications are commonly used to make informed business decisions.  As businesses migrate workloads to the cloud, adoption of BI software continues to increase. Business intelligence focuses narrowly on what is happening in the business.   While business analytics more broadly includes solutions that help you leverage that insight to plan for the future. Business intelligence uses descriptive analytics to formulate conclusions about historical and current performance, providing context around changes in key performance indicators (KPIs).

BI platforms are have dash-boarding, ad hoc reporting and data visualization capabilities At the core, BI rely on data warehouses, ETL, and OLAP.   After data is pre-processed and aggregated, it is fed into one central repository, such as a data warehouse or data mart, which supports business analytics and reporting tools. For larger data sets, businesses typically use an open source data storage framework known as Apache Hadoop.  BI solutions rely on a data integration process that combines data from multiple data sources into a single, consistent data store that is loaded into a data warehouse or other target system. ETL is the three steps in this process, which are extract, transform and load.  

OLAP is an acronym which stands for online analytical processing. This technology extracts big data from relational tables and reorganizes it into a multidimensional format, enabling fast processing and insightful data analysis.

Avant-Grade-Technologies used all of these technologies in DevOps and with BigData .   The objective is to use BigData to and Structured Data for decision making.  All of these  systems are automated and integrated in other business systems, such as marketing, finance, engineering and human resources.  All of these methods are used together to provide a holistic solution and improve decision making.