This robot did all the things we expect a good robot to do: it was dexterous, precise and even creative. These techniques are closely intertwined with the mechanics, sensors and actuators of the robot.

## Intelligence and embodiment

A flywheel that rotates orthogonally to the robot's motion induces a right turn as soon as it hits the ground when the front steering wheel goes over the edge.

## A roboticists’ problem

You can use a technique known as depth-first search to do this, but must not only map the environment but also locate within it, e.g. by recognizing places and dead reckoning on the map. Once you've found the gold, plan the shortest route back to the exit, which you can then use to go back and forth until all the gold is harvested.

## Ratslife: an example of autonomous mobile robotics

Since the time allowed to do this is limited, there is a good chance that the robot's energy will soon run out. From a strategic point of view, it can make sense to wait just in front of the feeder and only approach it shortly before the robot runs out of power.

Autonomous mobile robots: some core challenges

## Autonomous manipulation: some core challenges

Propulsion, the focus of this chapter, is the robot's ability to move and manipulate the world. In particular, we distinguish locomotion as the robot's ability to move itself, and manipulation as the robot's ability to move objects in the environment.

## Locomotion and manipulation examples

Regardless of whether the robot is rolling or walking, the dominant actuator type is rotary. Another type of mechanism is the prismatic or linear joint (see Figure 2.5 for an example) that allows the robot to stretch and contract a link.

## Static and dynamic stability

Grasping is a hard problem in itself and is therefore dealt with in its own chapter (Chapter 5). This type of joint is usually combined with rotary joints and allows, for example, a robot arm to move up and down, or a robot walker to extend or retract its leg.

## Degrees of freedom

In such a scenario, the degrees of freedom of the end effector are limited to moving up and down, sideways and rotating. A precise definition of the number of degrees of freedom is difficult and requires the derivation of analytical expressions for the end-effector position and orientation, which will be the subject of Chapter 3.

## Coordinate Systems and Frames of Reference

To describe the orientation of a point, we will associate a coordinate system with it. We have now established how to express the orientation of a coordinate system using a rotation matrix.

## Forward Kinematics

Knowing the robot's kinematic properties (e.g. the length of all rigid joints, similarly tol1 and l2 in Figure 3.1), the position of its end effector is fully described by its joint configuration (angle for rotary joints, length for prismatic joints). The angle α between the z-axes of the two links in relation to the common normal (link twist), i.e.

## Inverse Kinematics

We will now look at the inverse kinematics of the 2-jointed arm, which we presented in Figure 3.1. Providing the orientation of the robot in addition to the desired position therefore allows solving for α and β only as a function of x, y and θ.

## Differential Kinematics

Here, we choose to center the coordinate system on the robot axis and align Rx with its default heading direction. Equation (3.33) only provides us with the relation between the speed of the robot's wheel and its speed in the inertial frame.

## Inverse Differential Kinematics

Positive angular velocity would result in backward movement, as defined by the robot's coordinate system. The forward and reverse kinematics of a mobile robot are performed with respect to the robot's velocity rather than its position.

## Statics

Mechanical power W is defined as force times velocity, which can be generalized as generalized forces times generalized velocities: W = FT ·ν. It can be interpreted as follows: to counteract an external wrench Fe =−F applied to the end effector by the environment in a static configuration q, the robot must apply torquesτ at its joints as specified in equation (4.5).

## Kineto-Statics Duality

In this case the desired wrench (assuming that our z− .-axis in Cartesian space is the vertical one and it points upwards) will be:.

## Manipulability

It is always positive and reaches a maximum when the ellipsoid is close to the sphere and the robot can move isotropically in any direction. This ellipsoid indicates the end-effector forces with which the robot can act on the environment in a given joint configuration q.

## The theory of grasping

If µ is low, the cone will be relatively narrow, requiring the force to be perpendicular to the object's surface to avoid slipping. The rim of the suction cup provides both friction and multiple contact points to prevent the object from slipping and constrain the object beyond the normal force applied by the vacuum.

## Simple grasping mechanisms

The control strategy for parallel jaw clamps requires an accurate estimation of the position of the object of interest and an accurate positioning of the clamp so that the object is exactly in the center of the two jaws. In addition, the control strategy is the same as for the parallel jaw clamp, requiring an accurate estimation of the object's position.

## Electric motors

Due to the shape of the coil, the magnetic fields of all the wires overlap and create a strong magnetic field in the center of the coil. This can be achieved by reversing the direction of the current flowing through the coil.

## Hydraulic and pneumatic actuators

These actuators usually consist of a flexible rubber material with an inner cavity that can be filled with air. From a kinematics perspective, an ideal, fully soft robot can be modeled as a platform with an infinite number of mechanical degrees of freedom.

## Safety considerations

These problems can be overcome by always-on braking mechanisms that act in the absence of power or by using gear ratios that are so high that the overall mechanism cannot be reversed. For example, localization can be achieved by measuring how many degrees a wheel has turned with a sensor known as an "encoder" (7.2), a sensor that can be implemented based on a wide variety of physical principles.

## Terminology

Instead, the accuracy of a sensor is given by the ratio of the range and statistical variance of the signal. Proprioception involves estimating the robot's joint angles, velocities, and internal torques and forces.

Sensors that measure the robot’s joint configuration

## Sensors that measure ego-motion

When the system is moved, the moment of inertia preserves the original orientation of the disk, making it possible to measure the orientation of the system relative to where the system was originally. Coriolis forces are best understood by moving orthogonally to the direction of rotation of a vinyl record player.

## Measuring force

Here, one link of the robot connects to the outer ring, the other to the inner hub. A pressure sensor is a device capable of detecting either a contact/collision as a binary data (in which case it is generally referred to as a touch sensor) or a pressure gradient applied to it.

## Sensors to measure distance

Because the transmitter knows the shape of the transmitted wave, it can calculate the phase difference between the transmitted and received signal. In practice, this is done by combining the receiver with a very fast electronic shutter that operates at the same frequency of the emitted light.

## Sensors to sense global pose

The laser scanner you are considering can detect up to this range and has a bandwidth of 10Hz. Assume 300m/s for the speed of sound in the following. What kind of information does the environment provide and what kind of sensor do you need to exploit it.

## Images as two-dimensional signals

Vision is one of the most information-rich sensor systems available to both humans and robots. The next sections will provide both an intuition of what kind of meaningful information is hidden in such abstract data and provide concrete examples of signal processing techniques that make this information emerge.

## From signals to information

The image of a Dalmatian dog can be clearly recognized by most onlookers, even though there are low-level clues such as edging present only for ears, chin, and parts of the legs. The outlines of the animals are marked in a flipped version of the image in the inset.

## Basic image operations

Doing this for all x and ally is equivalent to physically "sliding" the filter g() along the image. Gaussian differences can also be used to approximate the Gaussian Laplacian, i.e. the sum of the second derivatives of a Gaussian kernel.

## Extracting Structure from Vision

Note that since the camera pair takes two images of the same scene, two projections in the corresponding image planes of the same point in the world can be correlated with each other to determine the point's 3D position. This expression, clearly induced by the geometry expressed in Figure 8.7, allows for the alternative solution of the values that make up the entries of the essential matrix and not those of the internal and external parameters of the camera.

## Computer Vision and Machine Learning

Use a convenient robot simulation environment that allows you to access a simulated camera in a world with simple features such as geometric shapes with different colors etc. Use simple geometric objects like a red ball and calculate their distance using stereo disparity.

## Feature detection as an information-reduction problem

In Chapter 8, we were introduced to convolution-based filters such as blurring, edge detection, or binary operations such as We are now interested in methods for extracting higher-level features such as lines and techniques for extracting them using these types of processing algorithms.

## Features

Therefore, the goal of most image processing algorithms is to first reduce the information content in a meaningful way and then extract the relevant information.

## Line recognition

A powerful solution to this problem is to randomly sample possible lines and retain those that meet a certain desired quality given that the number of points is somewhat close to the best fit. Random lines are evaluated by counting the number of points close ("inliers"), darker lines are better fits.

## Scale-invariant feature transforms

One approach to this is to find all features in an image and compare them to those in the database. This allows each detected feature to "vote" for the location of the object it is most closely associated with in the database.

## Feature detection and machine learning

Artificial neural networks (ANNs) are part of a class of machine learning techniques loosely inspired by neural operations in the human brain. Not so long ago, neural networks were considered "deep" if they only had more than two layers.

## The simple Perceptron

For each of these applications, it is important to understand when deep learning can outperform traditional approaches, and when it cannot. In a nutshell, deep learning models become first choice when not enough information exists to model a system using first principles.

## Activation Functions

Typical activation functions used in neural networks: the sigmoid activation function, left, and the rectified linear unit (ReLU), right. The sigmoid function is attractive for learning as the direction in which the weights must move to improve the error is very clear near wx= 0 and it is quite simple to calculate its derivative.

## From the simple perceptron to Multi-layer neural net- works

Notation used to index weights (left) with respect to layerk and the multilayer network from Figure 10.5 (right). ˆyi−yi)2 (10.11) between the output ˆyi that the neural network with parameterswcomputes and the known value yi from the training set is minimized.

## From single outputs to higher dimensional data

While OHE transforms the training input into a discrete probability distribution, nothing in the neural network will ensure that the data will also come out that way. However, the "brutal" cutoff introduced by the arg max function is exactly what we want for the network to optimally match the training input.

## Objective functions and optimization

In any case, the entropy function has some interesting properties: first, the logarithm from 0 (negative infinity) to 1 is negative (thus giving probabilities positive values). H(y)−H(ˆy, y This difference between the entropy of the true distribution and the cross-entropy between the true and the estimated distribution is known as the Kullback-Leibler divergence.

## Convolutional Neural Networks

We also assume that the convolution kernel inputs will be indexed in a similar way. Note that the feature map “weights”—that is, the entries of the kernel matrix—are identical for each neuron in the feature map.

## Recurrent Neural Networks

To overcome this limitation, it is useful to introduce the notion of state in a neural network. An example of a recurrent neural network (left) and its extended version (right) looking back four time steps.