Support `@differentiable` function conversion for `init` references, in
addition to `func` references and literal closures. Minor usability improvement.
Resolves SR-12562.
Lift temporary cross-file derivative registration restriction.
`@derivative` attribute type-checking simplications coming soon: TF-1099.
Original function and derivative function must have same access level, with one
exception: public original functions may have internal `@usableFromInline`
derivatives.
Previously, two conditions were necessary to enable differentiable programming:
- Using the `-enable-experimental-differentiable-programming` frontend flag.
- Importing the `_Differentiation` module.
Importing the `_Differentiation` module is the true condition because it
contains the required compiler-known `Differentiable` protocol. The frontend
flag is redundant and cumbersome.
Now, the frontend flag is removed.
Importing `_Differentiation` is the only condition.
The `@transpose` attribute registers a function as the transpose of another
function-like declaration: a `func`, `init`, `subscript`, or `var` computed
property declaration.
The `@transpose` attribute also has an optional `wrt:` clause specifying the
linearity parameters, i.e. the parameters that are transposed with respect to.
The linearity parameters must conform to the `Differentiable` protocol and
satisfy `Self == TangentVector`.
If the `wrt:` clause is unspecified, the linearity parameters are inferred to be
all parameters that conform to `Differentiable` and that satisfy
`Self == TangentVector`.
`@transpose` attribute type-checking verifies that the type of the transpose
function declaration is consistent with the type of the referenced original
declaration and the linearity parameters.
Resolves TF-830.
Add `AdditiveArithmetic` derived conformances for structs and classes, gated by
the `-enable-experimental-differentiable-programming` flag.
Structs and classes whose stored properties all conform to `Differentiable` can
derive `Differentiable`:
- `associatedtype TangentVector: Differentiable & AdditiveArithmetic`
- Member `TangentVector` structs are synthesized whose stored properties are
all `var` stored properties that conform to `Differentiable` and that are
not `@noDerivative`.
- `mutating func move(along: TangentVector)`
The `@noDerivative` attribute may be declared on stored properties to opt out of
inclusion in synthesized `TangentVector` structs.
Some stored properties cannot be used in `TangentVector` struct synthesis and
are implicitly marked as `@noDerivative`, with a warning:
- `let` stored properties.
- These cannot be updated by `mutating func move(along: TangentVector)`.
- Non-`Differentiable`-conforming stored properties.
`@noDerivative` also implies `@_semantics("autodiff.nonvarying")`, which is
relevant for differentiable activity analysis.
Add type-checking and SILGen tests.
Resolves TF-845.
Add the `@differentiable` function conversion pipeline:
- New expressions that convert between `@differentiable`,
`@differentiable(linear)`, and non-`@differentiable` functions:
- `DifferentiableFunction`
- `LinearFunction`
- `DifferentiableFunctionExtractOriginal`
- `LinearFunctionExtractOriginal`
- `LinearToDifferentiableFunction`
- All the AST handling (e.g. printing) necessary for those expressions.
- SILGen for those expressions.
- CSApply code that inserts these expressions to implicitly convert between
the various function types.
- Sema tests for the implicit conversions.
- SILGen tests for the SILGen of these expressions.
Resolves TF-833.
Add type checking for `@differentiable` function types:
- Check that parameters and results conform to `Differentiable`.
- Implicitly conform parameters and results whose types are generic parameters
to `Differentiable`.
- Upstream most of the differentiable_func_type_type_checking.swift test from
`tensorflow` branch. A few function conversion tests have not been added
because they depend on the `@differentiable` function conversion pipeline.
Diagnose gracefully when the `Differentiable` protocol is unavailable because
`_Differentiation` has not been imported.
Resolves TF-823 and TF-1219.
Add `AdditiveArithmetic` derived conformances for structs, gated by the
`-enable-experimential-additive-arithmetic-derivation` flag.
Structs whose stored properties all conform to `AdditiveArithmetic` can derive
`AdditiveArithmetic`:
- `static var zero: Self`
- `static func +(lhs: Self, rhs: Self) -> Self`
- `static func -(lhs: Self, rhs: Self) -> Self`
- An "effective memberwise initializer":
- Either a synthesized memberwise initializer or a user-defined initializer
with the same type.
Effective memberwise initializers are used only by derived conformances for
`Self`-returning protocol requirements like `AdditiveArithmetic.+`, which
require memberwise initialization.
Resolves TF-844.
Unblocks TF-845: upstream `Differentiable` derived conformances.
Previously, all witnesses of a `@differentiable` protocol requirement were
required to have the same attribute (or one with superset parameter indices).
However, this leads to many annotations on witnesses and is not ideal for
usability. `@differentiable` attributes are really only significant on
public witnesses, so that they are clearly `@differentiable` at a glance (in
source code, interface files, and API documentation), without looking through
protocol conformance hierarchies.
Now, only *public* witnesses of `@differentiable` protocol requirements are
required to have the same attribute (or one with superset parameter indices).
For less-visible witnesses, an implicit `@differentiable` attribute is created
with the same configuration as the requirement's.
Resolves TF-1117.
Upstreams #29771 from tensorflow branch.
Delete `@differentiable` attribute `jvp:` and `vjp:` arguments for derivative
registration. `@derivative` attribute is now the canonical way to register
derivatives.
Resolves TF-1001.
Add test/AutoDiff/lit.local.cfg: run tests only when `differentiable_programming`
is enabled in lit. With this, individual tests no longer need
`REQUIRES: differentiable_programming`.
Move multi-functionality SIL tests from test/AutoDiff/SIL/Serialization to
test/AutoDiff/SIL.
Garden test filenames.
Semantically, an `inout` parameter is both a parameter and a result.
`@differentiable` and `@derivative` attributes now support original functions
with one "semantic result": either a formal result or an `inout` parameter.
Derivative typing rules for functions with `inout` parameters are now defined.
The differential/pullback type of a function with `inout` differentiability
parameters also has `inout` parameters. This is ideal for performance.
Differential typing rules:
- Case 1: original function has no `inout` parameters.
- Original: `(T0, T1, ...) -> R`
- Differential: `(T0.Tan, T1.Tan, ...) -> R.Tan`
- Case 2: original function has a non-wrt `inout` parameter.
- Original: `(T0, inout T1, ...) -> Void`
- Differential: `(T0.Tan, ...) -> T1.Tan`
- Case 3: original function has a wrt `inout` parameter.
- Original: `(T0, inout T1, ...) -> Void`
- Differential: `(T0.Tan, inout T1.Tan, ...) -> Void`
Pullback typing rules:
- Case 1: original function has no `inout` parameters.
- Original: `(T0, T1, ...) -> R`
- Pullback: `R.Tan -> (T0.Tan, T1.Tan, ...)`
- Case 2: original function has a non-wrt `inout` parameter.
- Original: `(T0, inout T1, ...) -> Void`
- Pullback: `(T1.Tan) -> (T0.Tan, ...)`
- Case 3: original function has a wrt `inout` parameter.
- Original: `(T0, inout T1, ...) -> Void`
- Pullback: `(inout T1.Tan) -> (T0.Tan, ...)`
Resolves TF-1164.
- Support `@differentiable` and `@derivative` attributes for original
initializers in final classes. Reject original initializers in non-final
classes.
- Synchronize tests.
Attempt to look up original function before checking whether the `value:` result
conforms to `Differentiable`.
This improves diagnostics: "original function not found" should be diagnosed as
early as possible.
Previously, `@derivative` attribute type-checking produced a confusing error
referencing unbound types `T` and `U`:
```
'@derivative(of:)' attribute requires function to return a two-element tuple of
type '(value: T..., pullback: (U.TangentVector) -> T.TangentVector...)' or
'(value: T..., differential: (T.TangentVector...) -> U.TangentVector)'
```
Now, the error is less confusing:
```
'@derivative(of:)' attribute requires function to return a two-element tuple;
first element must have label 'value:' and second element must have label
'pullback:' or 'differential:'
```
For protocol requirements and class members with `@differentiable` attribute,
conforming types and subclasses must have the same `@differentiable` attribute
(or one with a superset of differentiability parameters) on implementing/
overriding declarations.
For implementing/overriding declarations that are missing a `@differentiable`
attribute, emit a fix-it that adds the missing attribute.
Resolves TF-1118.
The `@differentiable` attribute marks a function as differentiable.
Example:
```
@differentiable(wrt: x, jvp: derivativeFoo where T: Differentiable)
func id<T>(_ x: T) -> T { x }
```
The `@differentiable` attribute has an optional `wrt:` clause specifying the
parameters that are differentiated "with respect to", i.e. the differentiability
parameters. The differentiability parameters must conform to the
`Differentiable` protocol.
If the `wrt:` clause is unspecified, the differentiability parameters are
currently inferred to be all parameters that conform to `Differentiable`.
The `@differentiable` attribute also has optional `jvp:` and `vjp:` labels
for registering derivative functions. These labels are deprecated in favor of
the `@derivative` attribute and will be removed soon.
The `@differentiable` attribute also has an optional `where` clause, specifying
extra differentiability requirements for generic functions.
The `@differentiable` attribute is gated by the
`-enable-experimental-differentiable-programming` flag.
Code changes:
- Add `DifferentiableAttributeTypeCheckRequest`.
- Currently, the request returns differentiability parameter indices, while
also resolving `JVPFunction`, `VJPFunction`, and
`DerivativeGenericSignature` and mutating them in-place in
`DifferentiableAttr`. This was the simplest approach that worked without
introducing request cycles.
- Add "is type-checked" bit to `DifferentiableAttr`.
- Alternatively, I tried changing `DifferentiableAttributeTypeCheckRequest` to
use `CacheKind::Cache` instead of `CacheKind::SeparatelyCached`, but it did
not seem to work: `@differentiable` attributes in non-primary-files were
left unchecked.
Type-checking rules (summary):
- `@differentiable` attribute must be declared on a function-like "original"
declaration: `func`, `init`, `subscript`, `var` (computed properties only).
- Parsed differentiability parameters must be valid (if they exist).
- Parsed `where` clause must be valid (if it exists).
- Differentiability parameters must all conform to `Differentiable`.
- Original result must all conform to `Differentiable`.
- If JVP/VJP functions are specified, they must match the expected type.
- `@differentiable(jvp:vjp:)` for derivative registration is deprecated in
favor of `@derivative` attribute, and will be removed soon.
- Duplicate `@differentiable` attributes with the same differentiability
parameters are invalid.
- For protocol requirements and class members with `@differentiable` attribute,
conforming types and subclasses must have the same `@differentiable` attribute
(or one with a superset of differentiability parameter indices) on
implementing/overriding declarations.
Enable qualified declaration names in `@derivative` attribute, just like
`@transpose` attribute.
`DerivativeAttr` now stores a base type `TypeRepr *`, which is non-null for
parsed attributes that reference a qualified original declaration.
Add `TypeResolutionFlags::AllowModule` flag to enable module lookup via
`TypeChecker::lookupMember` given a `ModuleType`.
Add tests for type-qualified and module-qualified declaration names.
Resolves TF-1058.
The new _Differentiable module is not available in any shipping OS release, but its public API currently doesn’t have availability, either.
Temporarily disable tests that import it when we’re testing with OS-provided libraries.
The typecheck test test/AutoDiff/stdlib/differentiable_protocol.swift imports _Differentable but still somehow succeeds in these configs, so leave that one enabled.
rdar://57975086
The `@derivative` attribute registers a function as a derivative of another
function-like declaration: a `func`, `init`, `subscript`, or `var` computed
property declaration.
The `@derivative` attribute also has an optional `wrt:` clause specifying the
parameters that are differentiated "with respect to", i.e. the differentiation
parameters. The differentiation parameters must conform to the `Differentiable`
protocol.
If the `wrt:` clause is unspecified, the differentiation parameters are inferred
to be all parameters that conform to `Differentiable`.
`@derivative` attribute type-checking verifies that the type of the derivative
function declaration is consistent with the type of the referenced original
declaration and the differentiation parameters.
The `@derivative` attribute is gated by the
`-enable-experimental-differentiable-programming` flag.
Resolves TF-829.
Adds parsing for a type attribute `@differentiable`, which is optionally allowed to have argument `@differentiable(linear)`.
The typechecker currently rejects all uses of `@differentiable` with "error: attribute does not apply to type". Future work (https://bugs.swift.org/browse/TF-871https://bugs.swift.org/browse/TF-873) will update the typechecker to allow this attribute in places where it is allowed.
Resolves https://bugs.swift.org/browse/TF-822.
