Add `Differentiable.withDerivative(_:)`, a "derivative surgery" API.
`Differentiable.withDerivative(_:)` is an identity function returning `self`.
It takes a closure and applies it to the derivative of the return value, in
contexts where the return value is differentiated with respect to.
JVP functions are forward-mode derivative functions. They take original
arguments and return original results and a differential function. Differential
functions take derivatives wrt arguments and return derivatives wrt results.
`JVPEmitter` is a cloner that emits JVP and differential functions at the same
time. In JVP functions, function applications are replaced with JVP function
applications. In differential functions, function applications are replaced
with differential function applications.
In JVP functions, each basic block takes a differential struct containing callee
differentials. These structs are consumed by differential functions.
The differentiation transform does the following:
- Canonicalizes differentiability witnesses by filling in missing derivative
function entries.
- Canonicalizes `differentiable_function` instructions by filling in missing
derivative function operands.
- If necessary, performs automatic differentiation: generating derivative
functions for original functions.
- When encountering non-differentiability code, produces a diagnostic and
errors out.
Partially resolves TF-1211: add the main canonicalization loop.
To incrementally stage changes, derivative functions are currently created
with empty bodies that fatal error with a nice message.
Derivative emitters will be upstreamed separately.
Previously, two conditions were necessary to enable differentiable programming:
- Using the `-enable-experimental-differentiable-programming` frontend flag.
- Importing the `_Differentiation` module.
Importing the `_Differentiation` module is the true condition because it
contains the required compiler-known `Differentiable` protocol. The frontend
flag is redundant and cumbersome.
Now, the frontend flag is removed.
Importing `_Differentiation` is the only condition.
* Add all [differential operators](https://github.com/apple/swift/blob/master/docs/DifferentiableProgramming.md#list-of-differential-operators).
* Add `withoutDerivative(at:)`, used for efficiently stopping the derivative propagation at a value and causing the derivative at the value to be zero.
* Add utility `differentiableFunction(from:)`, used for creating a `@differentiable` function from an original function and a derivative function.
Mostly work done by @marcrasi and @dan-zheng.
Partially resolves TF-843.
TODO:
* Add `AnyDerivative`.
* Add `Array.differentiableMap(_:)` and `differentiableReduce(_:_:)`.
Add `Differentiable` conformances for floating-point types to the
`_Differentiation` module. The `TangentVector` associated type for
floating-point types is `Self`.
This design adheres to the differentiable programming manifesto:
docs/DifferentiableProgramming.md.
Partially resolves TF-1052.
The `_Differentiation` module is the experimental support library for
differentiable programming. It is built when the build-script flag
`--enable-experimental-differentiable-programming` is enabled.
The `Differentiable` protocol generalizes all types that work with
differentiation. It is a core piece of the differentiable programming
project. Other parts depending on the `Differentiable` protocol will
be upstreamed piece by piece.
The `Differentiable` protocol is compiler-known and will be used during
type-checking, SILGen, and the SIL differentiation transform.
