This commit is contained in:
2025-05-13 01:34:53 +03:00
parent 427735e23d
commit 83f3f1c7d4
945 changed files with 633484 additions and 0 deletions
@@ -0,0 +1,4 @@
__pycache__
.Trash*
/build
/.tup
@@ -0,0 +1,127 @@
## Overview ##
The goal of Fibre is to provide a framework to make suckless distributed applications easier to program.
In particular:
- Nobody likes boiler plate code. Using a remote object should feel almost
exactly as if it was local. No matter if it's in a different process, on
a USB device, connected via Bluetooth, over the Internet or all at the
same time.
All complexity arising from the system being distributed should be taken
care of by Fibre while still allowing the application developer to easily
fine-tune things.
- Fibre has the ambition to run on most major platforms and provide bindings
for the most popular languages. Even bare metal embedded systems with very
limited resources. See the Compatibility section for the current status.
- Once you deployed your application and want to change the interface, don't
worry about breaking other applications. With Fibre's object model, the
most common updates like adding methods, properties or arguments won't
break anything. Sometimes you can get away with removing methods if they
weren't used by other programs. **This is not implemented yet.**
## Current Status ##
The project is in an early stage and the focus so far was to get a minimum working implementation.
* **C++**: Currently only supports the server side (i.e. publishing local objects). The C++ library comes with builtin support for TCP and UDP transport layers on Posix platforms. The library can easily be used with user provided transport layers.
* **Python**: Currently only supports the client side (i.e. using remote objects). The Python library comes with builtin support for TCP, UDP, USB and UART transport layers.
Support for more languages (most importantly JavaScript) will be added once the protocol matures. Feel free to add your contribution.
## Show me some code
Consider this program:
```
class TestClass {
public:
float property1;
float property2;
float set_both(float arg1, float arg2) {
property1 = arg1;
property2 = arg2;
return property1 + property2;
}
};
int main() {
TestClass test_object = TestClass();
while (1) {
printf("test_object.property1: %f\n", test_object.property1);
usleep(1000000 / 5); // 5 Hz
}
}
```
Say you want to publish `test_object` so that a remote Fibre node can use it.
1. Add includes
```C++
#include <fibre/protocol.hpp>
#include <fibre/posix_tcp.hpp>
```
1. Add Fibre export definitions to the exported class
```C++
class TestClass {
[...]
FIBRE_EXPORTS(TestClass,
make_protocol_property("property1", &property1),
make_protocol_property("property2", &property2),
make_protocol_function("set_both", *obj, &TestClass::set_both, "arg1", "arg2")
);
};
```
Note: in the future this will be generated from a YAML file using automatic code generation.
1. Publish the object on Fibre
```C++
auto definitions = test_object.fibre_definitions;
fibre_publish(definitions);
```
Note: currently you must publish all objects at once. This will be fixed in the future.
1. Start the TCP server
```C++
std::thread server_thread_tcp(serve_on_tcp, 9910);
```
Note: this step will be replaced by a simple `fibre_start()` call in the future. All builtin transport layers then will be started automatically.
## Adding Fibre to your project ##
We recommend Git subtrees if you want to include the Fibre source code in another project.
Other contributors don't need to know anything about subtrees, to them the Fibre repo will be like any other normal directory.
#### Adding the repo
```
git remote add fibre-origin git@github.com:samuelsadok/fibre.git
git fetch fibre-origin
git subtree add --prefix=fibre --squash fibre-origin master
```
Instead of using the upstream remote, you might want to use your own fork for greater flexibility.
#### Pulling updates from upstream
```
git subtree pull --prefix=fibre --squash fibre-origin master
```
#### Contributing changes back to upstream
This requires push access to `fibre-origin`.
```
git subtree push --prefix=fibre fibre-origin master
```
## Projects using Fibre ##
- [ODrive](https://github.com/madcowswe/ODrive): High performance motor control
- [lightd](https://github.com/samuelsadok/lightd): Service that can be run on a Raspberry Pi (or similar) to control RGB LED strips
## Contribute ##
This project losely adheres to the [Google C++ Style Guide](https://google.github.io/styleguide/cppguide.html).
@@ -0,0 +1,90 @@
/*[# This is the original template, thus the warning below does not apply to this file #]
* ============================ WARNING ============================
* ==== This is an autogenerated file. ====
* ==== Any changes to this file will be lost when recompiling. ====
* =================================================================
*
* This file contains the toplevel handler for Fibre v0.1 endpoint operations.
*
* This endpoint-oriented approach will be deprecated in Fibre v0.2 in favor of
* a function-oriented approach and a more powerful object model.
*
*/
#ifndef __FIBRE_INTERFACES_HPP
#define __FIBRE_INTERFACES_HPP
#include <fibre/introspection.hpp>
// Note: with -Og the functions with large switch statements reserves a huge amount
// of stack space because they reserves separate space for the stack frame of each
// of the inlined functions.
// The minimum known set of flags to prevent this is `-O1 -fipa-sra`.
// `-O2`, `-O3` and `-Os` are supersets of this.
#pragma GCC push_options
#pragma GCC optimize ("s")
namespace fibre {
const unsigned char embedded_json[] = [[embedded_endpoint_definitions | to_c_string]];
const size_t embedded_json_length = sizeof(embedded_json) - 1;
const uint16_t json_crc_ = calc_crc16<CANONICAL_CRC16_POLYNOMIAL>(PROTOCOL_VERSION, embedded_json, embedded_json_length);
const uint32_t json_version_id_ = (json_crc_ << 16) | calc_crc16<CANONICAL_CRC16_POLYNOMIAL>(json_crc_, embedded_json, embedded_json_length);
static void get_property(Introspectable& result, size_t idx) {
switch (idx) {
[%- for endpoint in endpoints %]
[%- if endpoint.function.name == 'exchange' and endpoint.in_bindings | list == ['obj'] %]
case [[endpoint.id]]: { [[(endpoint.in_bindings['obj'] + '$') | replace(')$', ', &result.storage_)')]]; result.type_info_ = &FibrePropertyTypeInfo<[[endpoint.function.in['obj'].type.c_name]]>::singleton; } break;
[%- endif %]
[%- endfor %]
default: break;
}
}
bool endpoint_handler(int idx, cbufptr_t* input_buffer, bufptr_t* output_buffer) {
//Introspectable property = get_property(idx);
//if property.is_valid()
switch (idx) {
[%- for endpoint in endpoints %]
[%- if (endpoint.function.name == 'exchange' or endpoint.function.name == 'read') and endpoint.in_bindings | list == ['obj'] %]
case [[endpoint.id]]: { return [[endpoint.function.fullname | to_snake_case]]([% for k, arg in endpoint.function.in.items() %][% if k in endpoint.in_bindings %]static_cast<[[arg.type.c_name]]>([[endpoint.in_bindings[k]]])[% else %]std::nullopt[% endif %], [% endfor %][% for k, arg in endpoint.function.out.items() %][% if k in endpoint.out_bindings %]static_cast<[[arg.type.c_name]]*>([[endpoint.out_bindings[k]]])[% else %]nullptr[% endif %], [% endfor %]input_buffer, output_buffer); } break;
[%- else %]
case [[endpoint.id]]: { return [[endpoint.function.fullname | to_snake_case]]([% for k, arg in endpoint.function.in.items() %][% if k in endpoint.in_bindings %]static_cast<[[arg.type.c_name]]>([[endpoint.in_bindings[k]]])[% else %]std::nullopt[% endif %], [% endfor %][% for k, arg in endpoint.function.out.items() %][% if k in endpoint.out_bindings %]static_cast<[[arg.type.c_name]]*>([[endpoint.out_bindings[k]]])[% else %]nullptr[% endif %], [% endfor %]input_buffer, output_buffer); } break;
[%- endif %]
[%- endfor %]
default: return false;
}
}
bool is_endpoint_ref_valid(endpoint_ref_t endpoint_ref) {
if (endpoint_ref.json_crc != json_crc_) {
return false;
}
switch (endpoint_ref.endpoint_id) {
[%- for endpoint in endpoints %]
case [[endpoint.id]]: return true;
[%- endfor %]
default: return false;
}
}
bool set_endpoint_from_float(endpoint_ref_t endpoint_ref, float value) {
if (endpoint_ref.json_crc != json_crc_) {
return false;
}
Introspectable property{};
get_property(property, endpoint_ref.endpoint_id);
const FloatSettableTypeInfo* type_info = dynamic_cast<const FloatSettableTypeInfo*>(property.get_type_info());
return type_info && type_info->set_float(property, value);
}
}
#pragma GCC pop_options
#endif // __FIBRE_INTERFACES_HPP
@@ -0,0 +1,40 @@
/*[# This is the original template, thus the warning below does not apply to this file #]
* ============================ WARNING ============================
* ==== This is an autogenerated file. ====
* ==== Any changes to this file will be lost when recompiling. ====
* =================================================================
*
* This file contains serializing/deserializing stubs for the functions defined
* in your interface file.
*
*/
#include <fibre/bufptr.hpp>
[% for intf in interfaces.values() %]
[% for func in intf.functions.values() %]
static inline bool [[func.fullname | to_snake_case]]([% for arg in func.in.values() %]std::optional<[[arg.type.c_name]]> in_[[arg.name]], [% endfor %][% for arg in func.out.values() %][[arg.type.c_name]]* out_[[arg.name]], [% endfor %]fibre::cbufptr_t* input_buffer, fibre::bufptr_t* output_buffer) {
[%- if func.in %]
bool success = [% for arg in func.in.values() %](in_[[arg.name]].has_value() || (in_[[arg.name]] = fibre::Codec<[[arg.type.c_name]]>::decode(input_buffer)).has_value()[% if arg.optional %] || true[% endif %])[% if not loop.last %]
&& [% endif %][% endfor %];
[%- else %]
bool success = true;
[%- endif %]
if (!success) {
return false;
}
[%- if func.implementation %]
[% if func.out %]std::tuple<[% for arg in func.out.values() %][[arg.type.c_name]][[', ' if not loop.last]][% endfor %]> ret = [% endif %][[func.implementation]]([% for arg in func.in.values() %]in_[[arg.name]][% if not arg.optional %].value()[% endif %][[', ' if not loop.last]][% endfor %]);
[%- else %]
[% if func.out %]std::tuple<[% for arg in func.out.values() %][[arg.type.c_name]][[', ' if not loop.last]][% endfor %]> ret = [% endif %]in_[[(func.in.values() | first).name]].value()->[[func.name]]([% for arg in func.in.values() | skip_first %]in_[[arg.name]][% if not arg.optional %].value()[% endif %][[', ' if not loop.last]][% endfor %]);
[%- endif %]
[%- if func.out %]
return [% for arg in func.out.values() %]((out_[[arg.name]] && ((*out_[[arg.name]] = std::get<[[loop.index0]]>(ret)), true)) || fibre::Codec<[[arg.type.c_name]]>::encode(std::get<[[loop.index0]]>(ret), output_buffer))[% if not loop.last %]
&& [% endif %][% endfor %];
[%- else %]
return true;
[%- endif %]
}
[% endfor %]
[% endfor %]
@@ -0,0 +1,93 @@
#ifndef __FIBRE_BUFPTR_HPP
#define __FIBRE_BUFPTR_HPP
namespace fibre {
static inline bool soft_assert(bool expr) { return expr; } // TODO: implement
/**
* @brief Holds a reference to a buffer and a length.
* Since this class implements begin() and end(), you can use it with many
* standard algorithms that operate on iterable objects.
*/
template<typename T>
struct generic_bufptr_t {
using iterator = T*;
using const_iterator = const T*;
generic_bufptr_t(T* begin, size_t length) : begin_(begin), end_(begin + length) {}
generic_bufptr_t(T* begin, T* end) : begin_(begin), end_(end) {}
generic_bufptr_t() : begin_(nullptr), end_(nullptr) {}
template<size_t I>
generic_bufptr_t(T (&begin)[I]) : generic_bufptr_t(begin, I) {}
generic_bufptr_t(const std::vector<std::remove_const_t<T>>& vector)
: generic_bufptr_t(vector.data(), vector.size()) {}
generic_bufptr_t(const generic_bufptr_t<std::remove_const_t<T>>& other)
: generic_bufptr_t(other.begin_, other.end_) {}
generic_bufptr_t& operator+=(size_t num) {
if (!soft_assert(num <= size())) {
num = size();
}
begin_ += num;
return *this;
}
generic_bufptr_t operator++(int) {
generic_bufptr_t result = *this;
*this += 1;
return result;
}
T& operator*() {
return *begin_;
}
generic_bufptr_t take(size_t num) const {
if (!soft_assert(num <= size())) {
num = size();
}
generic_bufptr_t result = {begin_, num};
return result;
}
generic_bufptr_t skip(size_t num, size_t* processed_bytes = nullptr) const {
if (!soft_assert(num <= size())) {
num = size();
}
if (processed_bytes)
(*processed_bytes) += num;
return {begin_ + num, end_};
}
size_t size() const {
return end_ - begin_;
}
bool empty() const {
return size() == 0;
}
T*& begin() { return begin_; }
T*& end() { return end_; }
T* const & begin() const { return begin_; }
T* const & end() const { return end_; }
T& front() const { return *begin(); }
T& back() const { return *(end() - 1); }
T& operator[](size_t idx) { return *(begin() + idx); }
T* begin_;
T* end_;
};
using cbufptr_t = generic_bufptr_t<const unsigned char>;
using bufptr_t = generic_bufptr_t<unsigned char>;
}
#endif // __FIBRE_BUFPTR_HPP
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,56 @@
#ifndef __CRC_HPP
#define __CRC_HPP
#include <stdint.h>
#include <limits.h>
// Calculates an arbitrary CRC for one byte.
// Adapted from https://barrgroup.com/Embedded-Systems/How-To/CRC-Calculation-C-Code
template<typename T, unsigned POLYNOMIAL>
static T calc_crc(T remainder, uint8_t value) {
constexpr T BIT_WIDTH = (CHAR_BIT * sizeof(T));
constexpr T TOPBIT = ((T)1 << (BIT_WIDTH - 1));
// Bring the next byte into the remainder.
remainder ^= (value << (BIT_WIDTH - 8));
// Perform modulo-2 division, a bit at a time.
for (uint8_t bit = 8; bit; --bit) {
if (remainder & TOPBIT) {
remainder = (remainder << 1) ^ POLYNOMIAL;
} else {
remainder = (remainder << 1);
}
}
return remainder;
}
template<typename T, unsigned POLYNOMIAL>
static T calc_crc(T remainder, const uint8_t* buffer, size_t length) {
while (length--)
remainder = calc_crc<T, POLYNOMIAL>(remainder, *(buffer++));
return remainder;
}
template<unsigned POLYNOMIAL>
static uint8_t calc_crc8(uint8_t remainder, uint8_t value) {
return calc_crc<uint8_t, POLYNOMIAL>(remainder, value);
}
template<unsigned POLYNOMIAL>
static uint16_t calc_crc16(uint16_t remainder, uint8_t value) {
return calc_crc<uint16_t, POLYNOMIAL>(remainder, value);
}
template<unsigned POLYNOMIAL>
static uint8_t calc_crc8(uint8_t remainder, const uint8_t* buffer, size_t length) {
return calc_crc<uint8_t, POLYNOMIAL>(remainder, buffer, length);
}
template<unsigned POLYNOMIAL>
static uint16_t calc_crc16(uint16_t remainder, const uint8_t* buffer, size_t length) {
return calc_crc<uint16_t, POLYNOMIAL>(remainder, buffer, length);
}
#endif /* __CRC_HPP */
@@ -0,0 +1,336 @@
#ifndef __DECODERS_HPP
#define __DECODERS_HPP
#include "protocol.hpp"
#include "crc.hpp"
#include "cpp_utils.hpp"
#include <utility>
/* Base classes --------------------------------------------------------------*/
// @brief Base class for stream based decoders.
// A stream based decoder is a decoder that processes arbitrary length data blocks.
class StreamDecoder : public StreamSink {
public:
// @brief Returns 0 if no error ocurred, otherwise a non-zero error code.
// Once process_bytes returned an error, subsequent calls to get_status must return the same error.
// If the decoder is in an error state, the behavior of get_expected_bytes and process_bytes is undefined.
virtual int get_status() = 0;
// @brief Returns the minimum number of bytes that are still needed to complete this decoder.
// If 0, the decoder is considered complete and any subsequent call to process_bytes must process
// exactly 0 bytes.
// process_bytes() must always process all provided bytes unless the decoder expects no more bytes
// afterwards
virtual size_t get_expected_bytes() = 0;
};
// @brief Base class for a decoder that is fed in a block-wise fashion.
// This base class is provided for convenience when implementing certain types of decoders.
// A StreamDecoder can be obtained from a BlockDecoder by using StreamDecoder_from_BlockDecoder.
template<unsigned BLOCKSIZE>
class BlockDecoder {
public:
typedef std::integral_constant<size_t, BLOCKSIZE> block_size;
virtual int get_status() = 0;
virtual size_t get_expected_blocks() = 0;
virtual int process_block(const uint8_t block[BLOCKSIZE]) = 0;
private:
};
// @brief Base class for a decoder that is fed in a byte-wise fashion
// This base class is provided for convenience when implementing certain types of decoders.
// A StreamDecoder can be obtained from a ByteDecoder by using StreamDecoder_from_ByteDecoder.
class ByteDecoder {
public:
virtual int get_status() = 0;
virtual size_t get_expected_bytes() = 0;
virtual int process_byte(uint8_t byte) = 0;
};
/* Converter classes ---------------------------------------------------------*/
// @brief Encapsulates a BlockDecoder to make it look like a StreamDecoder
// @tparam T The encapsulated BlockDecoder type.
// Must inherit from BlockDecoder.
template<typename T, ENABLE_IF(TypeChecker<T>::template all_are<BlockDecoder<T::block_size::value>>())>
class StreamDecoder_from_BlockDecoder : public StreamDecoder {
public:
// @brief Imitates the constructor signature of the encapsulated type.
template<typename ... Args, ENABLE_IF(TypeChecker<Args...>::template first_is_not<StreamDecoder_from_BlockDecoder>())>
explicit StreamDecoder_from_BlockDecoder(Args&& ... args)
: block_decoder_(std::forward<Args>(args)...) {
EXPECT_TYPE(T, BlockDecoder<T::block_size::value>);
}
inline int get_status() final {
return block_decoder_.get_status();
}
inline size_t get_expected_bytes() final {
size_t expected_bytes = block_decoder_.get_expected_blocks() * T::block_size::value;
return expected_bytes - std::min(expected_bytes, buffer_pos_);
}
inline int process_bytes(const uint8_t* buffer, size_t length, size_t* processed_bytes) final {
while (!get_status() && get_expected_bytes() && length) {
// use the incoming bytes to fill internal buffer to get a complete block
size_t n_copy = std::min(length, T::block_size::value - buffer_pos_);
memcpy(buffer_ + buffer_pos_, buffer, n_copy);
buffer += n_copy;
length -= n_copy;
if (processed_bytes) (*processed_bytes) += n_copy;
buffer_pos_ += n_copy;
// if we have a full block, process it
if (buffer_pos_ == T::block_size::value) {
block_decoder_.process_block(buffer_);
buffer_pos_ = 0;
}
}
return get_status();
}
size_t get_free_space() { return SIZE_MAX; } // TODO: deprecate
private:
T block_decoder_;
size_t buffer_pos_ = 0;
uint8_t buffer_[T::block_size::value];
};
// @brief Encapsulates a ByteDecoder to make it look like a BlockDecoder
// @tparam T The encapsulated ByteDecoder type.
// Must inherit from ByteDecoder.
template<typename T, ENABLE_IF(TypeChecker<T>::template all_are<ByteDecoder>())>
class BlockDecoder_from_ByteDecoder : public BlockDecoder<1> {
public:
// @brief Imitates the constructor signature of the encapsulated type.
template<typename ... Args, ENABLE_IF(TypeChecker<Args...>::template first_is_not<BlockDecoder_from_ByteDecoder>())>
BlockDecoder_from_ByteDecoder(Args&& ... args)
: byte_decoder_(std::forward<Args>(args)...) {
EXPECT_TYPE(T, ByteDecoder);
}
inline int get_status() final {
return byte_decoder_.get_status();
}
inline size_t get_expected_blocks() final {
return byte_decoder_.get_expected_bytes();
}
inline int process_block(const uint8_t block[1]) final {
int status = byte_decoder_.process_byte(*block);
return status;
}
private:
T byte_decoder_;
};
// @brief Encapsulates a ByteDecoder to make it look like a StreamDecoder
// @tparam T The encapsulated ByteDecoder type.
// Must inherit from ByteDecoder.
template<typename T, ENABLE_IF(TypeChecker<T>::template all_are<ByteDecoder>())>
class StreamDecoder_from_ByteDecoder : public StreamDecoder {
public:
// @brief Imitates the constructor signature of the encapsulated type.
template<typename ... Args, ENABLE_IF(TypeChecker<Args...>::template first_is_not<StreamDecoder_from_ByteDecoder>())>
StreamDecoder_from_ByteDecoder(Args&& ... args)
: byte_decoder_(std::forward<Args>(args)...) {
EXPECT_TYPE(T, ByteDecoder);
}
inline int get_status() final {
return byte_decoder_.get_status();
}
inline size_t get_expected_bytes() final {
return byte_decoder_.get_expected_bytes();
}
inline size_t get_free_space() { return SIZE_MAX; }
inline int process_bytes(const uint8_t* buffer, size_t length, size_t* processed_bytes) final {
while (!byte_decoder_.get_status() && byte_decoder_.get_expected_bytes() && length) {
length--;
if (processed_bytes) (*processed_bytes)++;
byte_decoder_.process_byte(*(buffer++));
}
return byte_decoder_.get_status();
}
private:
T byte_decoder_;
};
/* Decoder implementations ---------------------------------------------------*/
template<typename T>
class VarintByteDecoder : public ByteDecoder {
public:
static constexpr T BIT_WIDTH = (CHAR_BIT * sizeof(T));
VarintByteDecoder(T& state_variable) :
state_variable_(state_variable)
{
}
size_t get_expected_bytes() final {
return done_ ? 0 : 1;
}
int get_status() final {
return status_;
}
int process_byte(uint8_t input_byte) final {
if (bit_pos_ == 0) {
LOG_FIBRE("start decoding varint, with 0x%02x => %zx\n", input_byte, (uintptr_t)&state_variable_);
state_variable_ = 0;
}
LOG_FIBRE("varint: decode %02x << %zu at %zx\n", input_byte, bit_pos_, &bit_pos_);
// we assume bit_pos_ < BIT_WIDTH
state_variable_ |= (static_cast<T>(input_byte & 0x7f) << bit_pos_);
if (((state_variable_ >> bit_pos_) & 0x7f) != static_cast<T>(input_byte & 0x7f)) {
LOG_FIBRE("varint overflow: tried to add %02x << %zu\n", input_byte, bit_pos_);
return (status_ = -1); // overflow
}
bit_pos_ += 7;
done_ = !(input_byte & 0x80);
return (status_ = (done_ || bit_pos_ < BIT_WIDTH) ? 0 : -1);
}
private:
T& state_variable_;
// At all times where status_ != 0 the following statement holds:
// (done_ || bit_pos_ < BIT_WIDTH)
//size_t bit_pos_ = 0; // bit position
size_t bit_pos_ = 0; // bit position
int status_ = 0;
bool done_ = false;
int data[1024] = {0};
};
template<typename T>
using VarintStreamDecoder = StreamDecoder_from_ByteDecoder<VarintByteDecoder<T>>;
// This double nested type should work identically but makes it way harder for the compiler to optimize
//template<typename T>
//using VarintBlockDecoder = BlockDecoder_from_ByteDecoder<VarintByteDecoder<T>>;
//template<typename T>
//using VarintStreamDecoder = StreamDecoder_from_BlockDecoder<VarintBlockDecoder<T>>;
template<typename T>
inline VarintStreamDecoder<T> make_varint_decoder(T& variable) {
return VarintStreamDecoder<T>(variable);
}
inline VarintStreamDecoder<GET_TYPE_OF(&ReceiverState::endpoint_id)> make_endpoint_id_decoder(ReceiverState& state) {
return make_varint_decoder(state.endpoint_id);
}
inline VarintStreamDecoder<GET_TYPE_OF(&ReceiverState::length)> make_length_decoder(ReceiverState& state) {
return make_varint_decoder(state.length);
}
template<uint8_t INIT, uint8_t POLYNOMIAL, typename TDecoder,
ENABLE_IF(TypeChecker<TDecoder>::template all_are<StreamDecoder>())>
class CRC8BlockDecoder : public BlockDecoder<CRC8_BLOCKSIZE> {
public:
CRC8BlockDecoder(TDecoder&& inner_decoder) :
inner_decoder_(std::forward<TDecoder>(inner_decoder)) {
}
int get_status() final {
return status_;
}
size_t get_expected_blocks() final {
return (inner_decoder_.get_expected_bytes() + CRC8_BLOCKSIZE - 2) / (CRC8_BLOCKSIZE - 1);
}
int process_block(const uint8_t input_block[4]) final {
current_crc_ = calc_crc8<POLYNOMIAL>(current_crc_, input_block, CRC8_BLOCKSIZE - 1);
if (current_crc_ != input_block[CRC8_BLOCKSIZE - 1])
return status_ = -1;
return status_ = inner_decoder_.process_bytes(input_block, CRC8_BLOCKSIZE - 1, nullptr);
}
private:
TDecoder inner_decoder_;
int status_ = 0;
uint8_t current_crc_ = INIT;
};
template<unsigned INIT, unsigned POLYNOMIAL, typename TDecoder>
using CRC8StreamDecoder = StreamDecoder_from_BlockDecoder<CRC8BlockDecoder<INIT, POLYNOMIAL, TDecoder>>;
template<unsigned INIT, unsigned POLYNOMIAL, typename TDecoder>
inline CRC8StreamDecoder<INIT, POLYNOMIAL, TDecoder> make_crc8_decoder(TDecoder&& decoder) {
return CRC8StreamDecoder<INIT, POLYNOMIAL, TDecoder>(std::forward<TDecoder>(decoder));
}
// TODO: ENABLE_IF(TypeChecker<TDecoders...>::template all_are<StreamDecoder>())
template<typename ... TDecoders>
class DecoderChain;
template<>
class DecoderChain<> : public StreamDecoder {
public:
size_t get_expected_bytes() { return 0; }
int get_status() { return 0; }
int process_bytes(const uint8_t *input, size_t length, size_t* processed_bytes) { return 0; }
size_t get_free_space() { return SIZE_MAX; } // TODO: deprecate
};
template<typename TDecoder, typename ... TDecoders>
class DecoderChain<TDecoder, TDecoders...> : public StreamDecoder {
public:
DecoderChain(TDecoder&& this_decoder, TDecoders&& ... subsequent_decoders) :
this_decoder_(std::forward<TDecoder>(this_decoder)),
subsequent_decoders_(std::forward<TDecoders>(subsequent_decoders)...)
{
EXPECT_TYPE(TDecoder, StreamDecoder);
}
int get_status() final {
// If this decoder or any of the subsequent decoders failed, return error code.
int this_status = this_decoder_.get_status();
int subsequent_status = subsequent_decoders_.get_status();
if (this_status)
return this_status;
else if (subsequent_status)
return subsequent_status;
else
return 0;
}
size_t get_expected_bytes() final {
return this_decoder_.get_expected_bytes() + subsequent_decoders_.get_expected_bytes();
}
int process_bytes(const uint8_t *input, size_t length, size_t* processed_bytes) final {
if (this_decoder_.get_expected_bytes()) {
LOG_FIBRE("decoder chain: process %zu bytes in segment %s\n", length, typeid(TDecoder).name());
size_t chunk = 0;
int status = this_decoder_.process_bytes(input, length, &chunk);
input += chunk;
length -= chunk;
if (processed_bytes) (*processed_bytes) += chunk;
if (status)
return status;
if (!length)
return 0;
}
return subsequent_decoders_.process_bytes(input, length, processed_bytes);
}
size_t get_free_space() { return SIZE_MAX; } // TODO: deprecate
private:
TDecoder this_decoder_;
DecoderChain<TDecoders...> subsequent_decoders_;
};
template<typename ... TDecoders>
inline DecoderChain<TDecoders...> make_decoder_chain(TDecoders&& ... decoders) {
return DecoderChain<TDecoders...>(std::forward<TDecoders>(decoders)...);
}
#endif // __DECODERS_HPP
@@ -0,0 +1,323 @@
#ifndef __ENCODERS_HPP
#define __ENCODERS_HPP
#include "protocol.hpp"
#include "crc.hpp"
#include "cpp_utils.hpp"
#include <utility>
struct Request {
endpoint_id_t endpoint_id;
size_t length;
};
/* Base classes --------------------------------------------------------------*/
// @brief Base class for all stream encoders
// A stream based encoder is an encoder that generates arbitrary length data blocks.
class StreamEncoder : public StreamSource {
public:
// @brief Returns 0 if no error ocurred, otherwise a non-zero error code.
// Once get_bytes returned an error, subsequent calls to get_status must return the same error.
// If the encoder is in an error state, the behavior of get_available_bytes and get_bytes is undefined.
virtual int get_status() = 0;
// @brief Returns the minimum number of bytes that will still be generated by this encoder.
// If 0, the encoder is considered complete and any subsequent call to get_bytes must generate
// exactly 0 bytes.
// get_bytes() must always generate as many bytes as requested unless the encoder generates no more bytes
// afterwards
virtual size_t get_available_bytes() = 0;
};
// @brief Base class for an encoder that is fed in a block-wise fashion.
// This base class is provided for convenience when implementing certain types of encoders.
// A StreamEncoder can be obtained from a BlockEncoder by using StreamEncoder_from_BlockEncoder.
template<unsigned BLOCKSIZE>
class BlockEncoder {
public:
typedef std::integral_constant<size_t, BLOCKSIZE> block_size;
virtual int get_status() = 0;
virtual size_t get_available_blocks() = 0;
virtual int get_block(uint8_t block[BLOCKSIZE]) = 0;
private:
};
// @brief Base class for an encoder that is fed in a byte-wise fashion
// This base class is provided for convenience when implementing certain types of encoders.
// A StreamEncoder can be obtained from a ByteEncoder by using StreamEncoder_from_ByteEncoder.
class ByteEncoder {
public:
virtual int get_status() = 0;
virtual size_t get_available_bytes() = 0;
virtual int get_byte(uint8_t *output_byte) = 0;
};
/* Converter classes ---------------------------------------------------------*/
// @brief Encapsulates a BlockEncoder to make it look like a StreamEncoder
// @tparam T The encapsulated BlockEncoder type.
// Must inherit from to BlockEncoder.
template<typename T, ENABLE_IF(TypeChecker<T>::template all_are<BlockEncoder<T::block_size::value>>())>
class StreamEncoder_from_BlockEncoder : public StreamEncoder {
public:
// @brief Imitates the constructor signature of the encapsulated type.
template<typename ... Args, ENABLE_IF(TypeChecker<Args...>::template first_is_not<StreamEncoder_from_BlockEncoder>())>
explicit StreamEncoder_from_BlockEncoder(Args&& ... args)
: block_encoder_(std::forward<Args>(args)...) {
EXPECT_TYPE(T, BlockEncoder<T::block_size::value>);
}
inline int get_status() final {
return buffered_bytes_ ? 0 : block_encoder_.get_status();
}
inline size_t get_available_bytes() final {
size_t available_bytes = block_encoder_.get_available_blocks() * T::block_size::value;
return available_bytes + buffered_bytes_;
}
inline int get_bytes(uint8_t* buffer, size_t length, size_t* generated_bytes) final {
while (!get_status() && get_available_bytes() && length) {
// if the buffer is empty, retrieve a new block from the encode
if (!buffered_bytes_) {
block_encoder_.get_block(buffer_);
buffered_bytes_ = T::block_size::value;
}
// hand the buffered bytes to the encoder
size_t n_copy = std::min(buffered_bytes_, length);
memcpy(buffer, buffer_ + T::block_size::value - n_copy, n_copy);
length -= n_copy;
buffer += n_copy;
if (generated_bytes) (*generated_bytes) += n_copy;
buffered_bytes_ -= n_copy;
}
return get_status();
}
private:
T block_encoder_;
size_t buffered_bytes_ = 0;
uint8_t buffer_[T::block_size::value];
};
// @brief Encapsulates a ByteEncoder to make it look like a BlockEncoder
// @tparam T The encapsulated ByteEncoder type.
// Must inherit from ByteEncoder.
template<typename T, ENABLE_IF(TypeChecker<T>::template all_are<ByteEncoder>())>
class BlockEncoder_from_ByteEncoder : public BlockEncoder<1> {
public:
// @brief Imitates the constructor signature of the encapsulated type.
template<typename ... Args, ENABLE_IF(TypeChecker<Args...>::template first_is_not<BlockEncoder_from_ByteEncoder>())>
BlockEncoder_from_ByteEncoder(Args&& ... args)
: byte_encoder_(std::forward<Args>(args)...) {
EXPECT_TYPE(T, ByteEncoder);
}
inline int get_status() final {
return byte_encoder_.get_status();
}
inline size_t get_available_blocks() final {
return byte_encoder_.get_available_bytes();
}
inline int get_block(uint8_t block[1]) final {
int status = byte_encoder_.get_byte(*block);
return status;
}
private:
T byte_encoder_;
};
// @brief Encapsulates a ByteEncoder to make it look like a StreamEncoder
// @tparam T The encapsulated ByteEncoder type.
// Must inherit from ByteEncoder.
template<typename T, ENABLE_IF(TypeChecker<T>::template all_are<ByteEncoder>())>
class StreamEncoder_from_ByteEncoder : public StreamEncoder {
public:
// @brief Imitates the constructor signature of the encapsulated type.
template<typename ... Args, ENABLE_IF(TypeChecker<Args...>::template first_is_not<StreamEncoder_from_ByteEncoder>())>
StreamEncoder_from_ByteEncoder(Args&& ... args)
: byte_encoder_(std::forward<Args>(args)...) {
EXPECT_TYPE(T, ByteEncoder);
}
inline int get_status() final {
return byte_encoder_.get_status();
}
inline size_t get_available_bytes() final {
return byte_encoder_.get_available_bytes();
}
inline int get_bytes(uint8_t* buffer, size_t length, size_t* generated_bytes) final {
while (!byte_encoder_.get_status() && byte_encoder_.get_available_bytes() && length) {
length--;
if (generated_bytes) (*generated_bytes)++;
byte_encoder_.get_byte(buffer++);
}
return byte_encoder_.get_status();
}
private:
T byte_encoder_;
};
/* Encoder implementations ---------------------------------------------------*/
template<typename T>
class VarintByteEncoder : public ByteEncoder {
public:
static constexpr T BIT_WIDTH = (CHAR_BIT * sizeof(T));
VarintByteEncoder(const T& state_variable) :
state_variable_(state_variable)
{}
size_t get_available_bytes() final {
return done_ ? 0 : 1;
}
int get_status() final {
return 0;
}
int get_byte(uint8_t *output_byte) final {
if (bit_pos_ == 0)
LOG_FIBRE("start encoding varint, from pos %d\n", bit_pos_);
*output_byte = (state_variable_ >> bit_pos_) & 0x7f;
bit_pos_ += 7;
if (bit_pos_ < BIT_WIDTH && (state_variable_ >> bit_pos_)) {
LOG_FIBRE("remainder: %x\n", state_variable_ >> bit_pos_);
*output_byte |= 0x80;
}else
done_ = true;
return 0;
}
private:
const T& state_variable_;
size_t bit_pos_ = 0; // bit position
int status_ = 0;
bool done_ = false;
};
template<typename T>
using VarintStreamEncoder = StreamEncoder_from_ByteEncoder<VarintByteEncoder<T>>;
template<typename T>
VarintStreamEncoder<T> make_varint_encoder(const T& variable) {
return VarintStreamEncoder<T>(variable);
}
VarintStreamEncoder<GET_TYPE_OF(&Request::endpoint_id)> make_endpoint_id_encoder(const Request& request) {
return make_varint_encoder(request.endpoint_id);
}
VarintStreamEncoder<GET_TYPE_OF(&Request::length)> make_length_encoder(const Request& request) {
return make_varint_encoder(request.length);
}
template<uint8_t INIT, uint8_t POLYNOMIAL, typename TEncoder,
ENABLE_IF(TypeChecker<TEncoder>::template all_are<StreamEncoder>())>
class CRC8BlockEncoder : public BlockEncoder<CRC8_BLOCKSIZE> {
public:
CRC8BlockEncoder(TEncoder&& inner_encoder)
: inner_encoder_(std::forward<TEncoder>(inner_encoder)) {}
int get_status() final {
return status_;
}
size_t get_available_blocks() final {
return (inner_encoder_.get_available_bytes() + CRC8_BLOCKSIZE - 2) / (CRC8_BLOCKSIZE - 1);
}
int get_block(uint8_t block[4]) final {
size_t generated_bytes = 0;
status_ = inner_encoder_.get_bytes(block, CRC8_BLOCKSIZE - 1, &generated_bytes);
if (status_)
return status_;
// zero out unused end of the block
while (generated_bytes < CRC8_BLOCKSIZE)
block[generated_bytes++] = 0;
block[CRC8_BLOCKSIZE - 1] = current_crc_ = calc_crc8<POLYNOMIAL>(current_crc_, block, CRC8_BLOCKSIZE - 1);
return 0;
}
private:
TEncoder inner_encoder_;
int status_ = 0;
uint8_t current_crc_ = INIT;
};
template<unsigned INIT, unsigned POLYNOMIAL, typename TEncoder>
using CRC8StreamEncoder = StreamEncoder_from_BlockEncoder<CRC8BlockEncoder<INIT, POLYNOMIAL, TEncoder>>;
template<unsigned INIT, unsigned POLYNOMIAL, typename TEncoder>
CRC8StreamEncoder<INIT, POLYNOMIAL, TEncoder> make_crc8_encoder(TEncoder&& encoder) {
return CRC8StreamEncoder<INIT, POLYNOMIAL, TEncoder>(std::forward<TEncoder>(encoder));
}
template<typename ... TEncoders>
class EncoderChain;
template<>
class EncoderChain<> : public StreamEncoder {
public:
size_t get_available_bytes() final { return 0; }
int get_status() final { return 0; }
int get_bytes(uint8_t *output, size_t length, size_t* generated_bytes) final { return 0; }
};
template<typename TEncoder, typename ... TEncoders>
class EncoderChain<TEncoder, TEncoders...> : public StreamEncoder {
public:
EncoderChain(TEncoder&& this_encoder, TEncoders&& ... subsequent_encoders) :
this_encoder_(std::forward<TEncoder>(this_encoder)),
subsequent_encoders_(std::forward<TEncoders>(subsequent_encoders)...)
{
EXPECT_TYPE(TEncoder, StreamEncoder);
}
size_t get_available_bytes() final {
return this_encoder_.get_available_bytes() + subsequent_encoders_.get_available_bytes();
}
int get_status() final {
// If this encoder or any of the subsequent encoders failed, return error code.
int this_status = this_encoder_.get_status();
int subsequent_status = subsequent_encoders_.get_status();
if (this_status)
return this_status;
else if (subsequent_status)
return subsequent_status;
else
return 0;
}
int get_bytes(uint8_t *output, size_t length, size_t* generated_bytes) final {
if (this_encoder_.get_available_bytes()) {
LOG_FIBRE("encoder chain: generate %zu bytes in segment %s\n", length, typeid(TEncoder).name());
size_t chunk = 0;
int status = this_encoder_.get_bytes(output, length, &chunk);
if (status)
return status;
output += chunk;
length -= chunk;
if (generated_bytes) *generated_bytes += chunk;
if (!length)
return 0;
}
return subsequent_encoders_.get_bytes(output, length, generated_bytes);
}
private:
TEncoder this_encoder_;
EncoderChain<TEncoders...> subsequent_encoders_;
};
template<typename ... TEncoders>
EncoderChain<TEncoders...> make_encoder_chain(TEncoders&& ... encoders) {
return EncoderChain<TEncoders...>(std::forward<TEncoders>(encoders)...);
}
#endif // __ENCODERS_HPP
@@ -0,0 +1,219 @@
#ifndef __FIBRE_INTROSPECTION_HPP
#define __FIBRE_INTROSPECTION_HPP
#include <stdlib.h>
#include <algorithm>
#include <cstring>
#pragma GCC push_options
#pragma GCC optimize ("s")
class TypeInfo;
class Introspectable;
using introspectable_storage_t = std::aligned_storage<16, 4>::type;
struct PropertyInfo {
const char * name;
const TypeInfo* type_info;
};
/**
* @brief Contains runtime accessible type information.
*
* Specifically, this information consists of a list of PropertyInfo items which
* enable accessing attributes of an object by a runtime string.
*
* Typically, for each combination of C++ type and Fibre interface implemented
* by this type, one (static constant) TypeInfo object will exist.
*/
class TypeInfo {
friend class Introspectable;
public:
TypeInfo(const PropertyInfo* property_table, size_t property_table_length)
: property_table_(property_table), property_table_length_(property_table_length) {}
virtual introspectable_storage_t get_child(introspectable_storage_t obj, size_t idx) const = 0;
Introspectable get_child(const Introspectable& obj, const char * name, size_t length) const;
protected:
template<typename T> static T& as(Introspectable& obj);
template<typename T> static const T& as(const Introspectable& obj);
template<typename T> static Introspectable make_introspectable(T obj, const TypeInfo* type_info);
private:
const PropertyInfo* property_table_;
size_t property_table_length_;
};
/**
* @brief Wraps a reference to an application object by attaching runtime
* accessible type information.
*
* The reference that is wrapped is typically a pointer but can also be a small
* temporary, on-demand constructed object such as a fibre::Property<...> which
* contains multiple pointers.
*/
class Introspectable {
friend class TypeInfo;
public:
Introspectable() {}
/**
* @brief Returns an Introspectable object for the attribute referenced by
* the specified attribute name.
*
* The name can consist of multiple parts separated by dots.
*
* If the attribute does not exist, an invalid Introspectable is returned.
*
* @param path: The name or path of the attribute.
* @param length: The maximum length of the name.
*/
Introspectable get_child(const char * path, size_t length) {
Introspectable current = *this;
const char * begin = path;
const char * end = std::find(begin, path + length, '\0');
while ((begin < end) && current.type_info_) {
const char * end_of_token = std::find(begin, end, '.');
current = current.get_direct_child(begin, end_of_token - begin);
begin = std::min(end, end_of_token + 1);
}
return current;
};
bool is_valid() {
return type_info_;
}
const TypeInfo* get_type_info() {
return type_info_;
}
private:
Introspectable get_direct_child(const char * name, size_t length) const {
for (size_t i = 0; i < type_info_->property_table_length_; ++i) {
if (!strncmp(name, type_info_->property_table_[i].name, length) && (length == strlen(type_info_->property_table_[i].name))) {
Introspectable result;
result.storage_ = type_info_->get_child(storage_, i);
result.type_info_ = type_info_->property_table_[i].type_info;
return result;
}
}
return {};
}
public: // these should technically be protected but are public for optimization reasons
// We use this storage to hold generic small objects. Usually that's a pointer
// but sometimes it's an on-demand constructed Property<...>.
// Caution: only put objects in here which are trivially copyable, movable
// and destructible as any custom operation wouldn't be called.
introspectable_storage_t storage_;
const TypeInfo* type_info_ = nullptr;
};
template<typename T> T& TypeInfo::as(Introspectable& obj) {
static_assert(sizeof(T) <= sizeof(obj.storage_));
return *(T*)&obj.storage_;
}
template<typename T> const T& TypeInfo::as(const Introspectable& obj) {
static_assert(sizeof(T) <= sizeof(obj.storage_));
return *(const T*)&obj.storage_;
}
template<typename T> Introspectable TypeInfo::make_introspectable(T obj, const TypeInfo* type_info) {
Introspectable introspectable;
as<T>(introspectable) = obj;
introspectable.type_info_ = type_info;
return introspectable;
}
// maybe_underlying_type_t<T> resolves to the underlying type of T if T is an enum type or otherwise to T itself.
template<typename T, bool = std::is_enum<T>::value> struct maybe_underlying_type;
template<typename T> struct maybe_underlying_type<T, true> { typedef std::underlying_type_t<T> type; };
template<typename T> struct maybe_underlying_type<T, false> { typedef T type; };
template<typename T> using maybe_underlying_type_t = typename maybe_underlying_type<T>::type;
struct StringConvertibleTypeInfo {
virtual bool get_string(const Introspectable& obj, char* buffer, size_t length) const { return false; }
virtual bool set_string(const Introspectable& obj, char* buffer, size_t length) const { return false; }
};
struct FloatSettableTypeInfo {
//virtual bool get_float(const Introspectable& obj, float* val) const { return false; }
virtual bool set_float(const Introspectable& obj, float val) const { return false; }
};
/* Built-in type infos ********************************************************/
template<typename T>
struct FibrePropertyTypeInfo;
// readonly property
template<typename T>
struct FibrePropertyTypeInfo<Property<const T>> : StringConvertibleTypeInfo, TypeInfo {
using TypeInfo::TypeInfo;
static const PropertyInfo property_table[];
static const FibrePropertyTypeInfo<Property<const T>> singleton;
introspectable_storage_t get_child(introspectable_storage_t obj, size_t idx) const override {
return {};
}
bool get_string(const Introspectable& obj, char* buffer, size_t length) const override {
return to_string(static_cast<maybe_underlying_type_t<T>>(as<const Property<const T>>(obj).read()), buffer, length, 0);
}
};
template<typename T>
const PropertyInfo FibrePropertyTypeInfo<Property<const T>>::property_table[] = {};
template<typename T>
const FibrePropertyTypeInfo<Property<const T>> FibrePropertyTypeInfo<Property<const T>>::singleton{FibrePropertyTypeInfo<Property<const T>>::property_table, sizeof(FibrePropertyTypeInfo<Property<const T>>::property_table) / sizeof(FibrePropertyTypeInfo<Property<const T>>::property_table[0])};
// readwrite property
template<typename T>
struct FibrePropertyTypeInfo<Property<T>> : FloatSettableTypeInfo, StringConvertibleTypeInfo, TypeInfo {
using TypeInfo::TypeInfo;
static const PropertyInfo property_table[];
static const FibrePropertyTypeInfo<Property<T>> singleton;
static const Introspectable make_introspectable(Property<T> obj) { return TypeInfo::make_introspectable(obj, &singleton); }
introspectable_storage_t get_child(introspectable_storage_t obj, size_t idx) const override {
return {};
}
bool get_string(const Introspectable& obj, char* buffer, size_t length) const override {
return to_string(static_cast<maybe_underlying_type_t<T>>(as<const Property<T>>(obj).read()), buffer, length, 0);
}
bool set_string(const Introspectable& obj, char* buffer, size_t length) const override {
maybe_underlying_type_t<T> value;
if (!from_string(buffer, length, &value, 0)) {
return false;
}
as<const Property<T>>(obj).exchange(static_cast<T>(value));
return true;
}
bool set_float(const Introspectable& obj, float val) const override {
maybe_underlying_type_t<T> value;
if (!conversion::set_from_float(val, &value)) {
return false;
}
as<const Property<T>>(obj).exchange(static_cast<T>(value));
return true;
}
};
template<typename T>
const PropertyInfo FibrePropertyTypeInfo<Property<T>>::property_table[] = {};
template<typename T>
const FibrePropertyTypeInfo<Property<T>> FibrePropertyTypeInfo<Property<T>>::singleton{FibrePropertyTypeInfo<Property<T>>::property_table, sizeof(FibrePropertyTypeInfo<Property<T>>::property_table) / sizeof(FibrePropertyTypeInfo<Property<T>>::property_table[0])};
#pragma GCC pop_options
#endif // __FIBRE_INTROSPECTION_HPP
@@ -0,0 +1,4 @@
#include "protocol.hpp"
int serve_on_tcp(unsigned int port);
@@ -0,0 +1,4 @@
#include "protocol.hpp"
int serve_on_udp(unsigned int port);
@@ -0,0 +1,621 @@
/*
see protocol.md for the protocol specification
*/
#ifndef __PROTOCOL_HPP
#define __PROTOCOL_HPP
// TODO: resolve assert
#define assert(expr)
#include <functional>
#include <limits>
#include <cmath>
//#include <stdint.h>
#include <stdio.h>
#include <string.h>
#include <unistd.h>
#include <cstring>
#include "crc.hpp"
#include "cpp_utils.hpp"
#include "bufptr.hpp"
#include "simple_serdes.hpp"
// Note that this option cannot be used to debug UART because it prints on UART
//#define DEBUG_FIBRE
#ifdef DEBUG_FIBRE
#define LOG_FIBRE(...) do { printf(__VA_ARGS__); } while (0)
#else
#define LOG_FIBRE(...) ((void) 0)
#endif
// Default CRC-8 Polynomial: x^8 + x^5 + x^4 + x^2 + x + 1
// Can protect a 4 byte payload against toggling of up to 5 bits
// source: https://users.ece.cmu.edu/~koopman/crc/index.html
constexpr uint8_t CANONICAL_CRC8_POLYNOMIAL = 0x37;
constexpr uint8_t CANONICAL_CRC8_INIT = 0x42;
constexpr size_t CRC8_BLOCKSIZE = 4;
// Default CRC-16 Polynomial: 0x9eb2 x^16 + x^13 + x^12 + x^11 + x^10 + x^8 + x^6 + x^5 + x^2 + 1
// Can protect a 135 byte payload against toggling of up to 5 bits
// source: https://users.ece.cmu.edu/~koopman/crc/index.html
// Also known as CRC-16-DNP
constexpr uint16_t CANONICAL_CRC16_POLYNOMIAL = 0x3d65;
constexpr uint16_t CANONICAL_CRC16_INIT = 0x1337;
constexpr uint8_t CANONICAL_PREFIX = 0xAA;
/* move to fibre_config.h ******************************/
typedef size_t endpoint_id_t;
struct ReceiverState {
endpoint_id_t endpoint_id;
size_t length;
uint16_t seqno_thread;
uint16_t seqno;
bool expect_ack;
bool expect_response;
bool enforce_ordering;
};
/*******************************************************/
constexpr uint16_t PROTOCOL_VERSION = 1;
// This value must not be larger than USB_TX_DATA_SIZE defined in usbd_cdc_if.h
constexpr uint16_t TX_BUF_SIZE = 32; // does not work with 64 for some reason
constexpr uint16_t RX_BUF_SIZE = 128; // larger values than 128 have currently no effect because of protocol limitations
// Maximum time we allocate for processing and responding to a request
constexpr uint32_t PROTOCOL_SERVER_TIMEOUT_MS = 10;
typedef struct {
uint16_t json_crc = 0;
uint16_t endpoint_id = 0;
} endpoint_ref_t;
namespace fibre {
// These symbols are defined in the autogenerated endpoints.hpp
extern const unsigned char embedded_json[];
extern const size_t embedded_json_length;
extern const uint16_t json_crc_;
extern const uint32_t json_version_id_;
bool endpoint_handler(int idx, cbufptr_t* input_buffer, bufptr_t* output_buffer);
bool endpoint0_handler(cbufptr_t* input_buffer, bufptr_t* output_buffer);
bool is_endpoint_ref_valid(endpoint_ref_t endpoint_ref);
bool set_endpoint_from_float(endpoint_ref_t endpoint_ref, float value);
}
template<typename T, typename = typename std::enable_if_t<!std::is_const<T>::value>>
inline size_t write_le(T value, uint8_t* buffer){
//TODO: add static_assert that this is still a little endian machine
std::memcpy(&buffer[0], &value, sizeof(value));
return sizeof(value);
}
template<typename T>
typename std::enable_if_t<std::is_const<T>::value, size_t>
write_le(T value, uint8_t* buffer) {
return write_le<std::remove_const_t<T>>(value, buffer);
}
template<>
inline size_t write_le<float>(float value, uint8_t* buffer) {
static_assert(CHAR_BIT * sizeof(float) == 32, "32 bit floating point expected");
static_assert(std::numeric_limits<float>::is_iec559, "IEEE 754 floating point expected");
uint32_t value_as_uint32;
std::memcpy(&value_as_uint32, &value, sizeof(uint32_t));
return write_le<uint32_t>(value_as_uint32, buffer);
}
template<typename T>
inline size_t read_le(T* value, const uint8_t* buffer){
// TODO: add static_assert that this is still a little endian machine
std::memcpy(value, buffer, sizeof(*value));
return sizeof(*value);
}
template<>
inline size_t read_le<float>(float* value, const uint8_t* buffer) {
static_assert(CHAR_BIT * sizeof(float) == 32, "32 bit floating point expected");
static_assert(std::numeric_limits<float>::is_iec559, "IEEE 754 floating point expected");
return read_le(reinterpret_cast<uint32_t*>(value), buffer);
}
// @brief Reads a value of type T from the buffer.
// @param buffer Pointer to the buffer to be read. The pointer is updated by the number of bytes that were read.
// @param length The number of available bytes in buffer. This value is updated to subtract the bytes that were read.
template<typename T>
static inline T read_le(const uint8_t** buffer, size_t* length) {
T result;
size_t cnt = read_le(&result, *buffer);
*buffer += cnt;
*length -= cnt;
return result;
}
class PacketSink {
public:
// @brief Get the maximum packet length (aka maximum transmission unit)
// A packet size shall take no action and return an error code if the
// caller attempts to send an oversized packet.
//virtual size_t get_mtu() = 0;
// @brief Processes a packet.
// The blocking behavior shall depend on the thread-local deadline_ms variable.
// @return: 0 on success, otherwise a non-zero error code
// TODO: define what happens when the packet is larger than what the implementation can handle.
virtual int process_packet(const uint8_t* buffer, size_t length) = 0;
};
class StreamSink {
public:
// @brief Processes a chunk of bytes that is part of a continuous stream.
// The blocking behavior shall depend on the thread-local deadline_ms variable.
// @param processed_bytes: if not NULL, shall be incremented by the number of
// bytes that were consumed.
// @return: 0 on success, otherwise a non-zero error code
virtual int process_bytes(const uint8_t* buffer, size_t length, size_t* processed_bytes) = 0;
// @brief Returns the number of bytes that can still be written to the stream.
// Shall return SIZE_MAX if the stream has unlimited lenght.
// TODO: deprecate
virtual size_t get_free_space() = 0;
/*int process_bytes(const uint8_t* buffer, size_t length) {
size_t processed_bytes = 0;
return process_bytes(buffer, length, &processed_bytes);
}*/
};
class StreamSource {
public:
// @brief Generate a chunk of bytes that are part of a continuous stream.
// The blocking behavior shall depend on the thread-local deadline_ms variable.
// @param generated_bytes: if not NULL, shall be incremented by the number of
// bytes that were written to buffer.
// @return: 0 on success, otherwise a non-zero error code
virtual int get_bytes(uint8_t* buffer, size_t length, size_t* generated_bytes) = 0;
// @brief Returns the number of bytes that can still be written to the stream.
// Shall return SIZE_MAX if the stream has unlimited lenght.
// TODO: deprecate
//virtual size_t get_free_space() = 0;
};
class StreamToPacketSegmenter : public StreamSink {
public:
explicit StreamToPacketSegmenter(PacketSink& output) :
output_(output)
{
};
int process_bytes(const uint8_t *buffer, size_t length, size_t* processed_bytes) override;
size_t get_free_space() { return SIZE_MAX; }
private:
uint8_t header_buffer_[3] = {0};
size_t header_index_ = 0;
uint8_t packet_buffer_[RX_BUF_SIZE] = {0};
size_t packet_index_ = 0;
size_t packet_length_ = 0;
PacketSink& output_;
};
class StreamBasedPacketSink : public PacketSink {
public:
explicit StreamBasedPacketSink(StreamSink& output) :
output_(output)
{
};
//size_t get_mtu() { return SIZE_MAX; }
int process_packet(const uint8_t *buffer, size_t length) override;
private:
StreamSink& output_;
};
// @brief: Represents a stream sink that's based on an underlying packet sink.
// A single call to process_bytes may result in multiple packets being sent.
class PacketBasedStreamSink : public StreamSink {
public:
explicit PacketBasedStreamSink(PacketSink& packet_sink) : _packet_sink(packet_sink) {}
~PacketBasedStreamSink() {}
int process_bytes(const uint8_t* buffer, size_t length, size_t* processed_bytes) override {
// Loop to ensure all bytes get sent
while (length) {
size_t chunk = length;
// send chunk as packet
if (_packet_sink.process_packet(buffer, chunk))
return -1;
buffer += chunk;
length -= chunk;
if (processed_bytes)
*processed_bytes += chunk;
}
return 0;
}
size_t get_free_space() { return SIZE_MAX; }
private:
PacketSink& _packet_sink;
};
// Implements the StreamSink interface by writing into a fixed size
// memory buffer.
class MemoryStreamSink : public StreamSink {
public:
MemoryStreamSink(uint8_t *buffer, size_t length) :
buffer_(buffer),
buffer_length_(length) {}
// Returns 0 on success and -1 if the buffer could not accept everything because it became full
int process_bytes(const uint8_t* buffer, size_t length, size_t* processed_bytes) override {
size_t chunk = length < buffer_length_ ? length : buffer_length_;
memcpy(buffer_, buffer, chunk);
buffer_ += chunk;
buffer_length_ -= chunk;
if (processed_bytes)
*processed_bytes += chunk;
return chunk == length ? 0 : -1;
}
size_t get_free_space() { return buffer_length_; }
private:
uint8_t * buffer_;
size_t buffer_length_;
};
// Implements the StreamSink interface by discarding the first couple of bytes
// and then forwarding the rest to another stream.
class NullStreamSink : public StreamSink {
public:
NullStreamSink(size_t skip, StreamSink& follow_up_stream) :
skip_(skip),
follow_up_stream_(follow_up_stream) {}
// Returns 0 on success and -1 if the buffer could not accept everything because it became full
int process_bytes(const uint8_t* buffer, size_t length, size_t* processed_bytes) override {
if (skip_ < length) {
buffer += skip_;
length -= skip_;
if (processed_bytes)
*processed_bytes += skip_;
skip_ = 0;
return follow_up_stream_.process_bytes(buffer, length, processed_bytes);
} else {
skip_ -= length;
if (processed_bytes)
*processed_bytes += length;
return 0;
}
}
size_t get_free_space() override { return skip_ + follow_up_stream_.get_free_space(); }
private:
size_t skip_;
StreamSink& follow_up_stream_;
};
// Implements the StreamSink interface by calculating the CRC16 checksum
// on the data that is sent to it.
class CRC16Calculator : public StreamSink {
public:
explicit CRC16Calculator(uint16_t crc16_init) :
crc16_(crc16_init) {}
int process_bytes(const uint8_t* buffer, size_t length, size_t* processed_bytes) override{
crc16_ = calc_crc16<CANONICAL_CRC16_POLYNOMIAL>(crc16_, buffer, length);
if (processed_bytes)
*processed_bytes += length;
return 0;
}
size_t get_free_space() override { return SIZE_MAX; }
uint16_t get_crc16() { return crc16_; }
private:
uint16_t crc16_;
};
namespace fibre {
template<typename T, typename = void>
struct Codec {
static std::optional<T> decode(cbufptr_t* buffer) { return std::nullopt; }
};
template<> struct Codec<bool> {
static std::optional<bool> decode(cbufptr_t* buffer) { return (buffer->begin() == buffer->end()) ? std::nullopt : std::make_optional((bool)*(buffer->begin()++)); }
static bool encode(bool value, bufptr_t* buffer) { return SimpleSerializer<uint8_t, false>::write(value, &(buffer->begin()), buffer->end()); }
};
template<> struct Codec<int8_t> {
static std::optional<int8_t> decode(cbufptr_t* buffer) { return SimpleSerializer<int8_t, false>::read(&(buffer->begin()), buffer->end()); }
static bool encode(int8_t value, bufptr_t* buffer) { return SimpleSerializer<int8_t, false>::write(value, &(buffer->begin()), buffer->end()); }
};
template<> struct Codec<uint8_t> {
static std::optional<uint8_t> decode(cbufptr_t* buffer) { return SimpleSerializer<uint8_t, false>::read(&(buffer->begin()), buffer->end()); }
static bool encode(uint8_t value, bufptr_t* buffer) { return SimpleSerializer<uint8_t, false>::write(value, &(buffer->begin()), buffer->end()); }
};
template<> struct Codec<int16_t> {
static std::optional<int16_t> decode(cbufptr_t* buffer) { return SimpleSerializer<int16_t, false>::read(&(buffer->begin()), buffer->end()); }
static bool encode(int16_t value, bufptr_t* buffer) { return SimpleSerializer<int16_t, false>::write(value, &(buffer->begin()), buffer->end()); }
};
template<> struct Codec<uint16_t> {
static std::optional<uint16_t> decode(cbufptr_t* buffer) { return SimpleSerializer<uint16_t, false>::read(&(buffer->begin()), buffer->end()); }
static bool encode(uint16_t value, bufptr_t* buffer) { return SimpleSerializer<uint16_t, false>::write(value, &(buffer->begin()), buffer->end()); }
};
template<> struct Codec<int32_t> {
static std::optional<int32_t> decode(cbufptr_t* buffer) { return SimpleSerializer<int32_t, false>::read(&(buffer->begin()), buffer->end()); }
static bool encode(int32_t value, bufptr_t* buffer) { return SimpleSerializer<int32_t, false>::write(value, &(buffer->begin()), buffer->end()); }
};
template<> struct Codec<uint32_t> {
static std::optional<uint32_t> decode(cbufptr_t* buffer) { return SimpleSerializer<uint32_t, false>::read(&(buffer->begin()), buffer->end()); }
static bool encode(uint32_t value, bufptr_t* buffer) { return SimpleSerializer<uint32_t, false>::write(value, &(buffer->begin()), buffer->end()); }
};
template<> struct Codec<int64_t> {
static std::optional<int64_t> decode(cbufptr_t* buffer) { return SimpleSerializer<int64_t, false>::read(&(buffer->begin()), buffer->end()); }
static bool encode(int64_t value, bufptr_t* buffer) { return SimpleSerializer<int64_t, false>::write(value, &(buffer->begin()), buffer->end()); }
};
template<> struct Codec<uint64_t> {
static std::optional<uint64_t> decode(cbufptr_t* buffer) { return SimpleSerializer<uint64_t, false>::read(&(buffer->begin()), buffer->end()); }
static bool encode(uint64_t value, bufptr_t* buffer) { return SimpleSerializer<uint64_t, false>::write(value, &(buffer->begin()), buffer->end()); }
};
template<> struct Codec<float> {
static std::optional<float> decode(cbufptr_t* buffer) {
std::optional<uint32_t> int_val = Codec<uint32_t>::decode(buffer);
return int_val.has_value() ? std::optional<float>(*reinterpret_cast<float*>(&int_val.value())) : std::nullopt;
}
static bool encode(float value, bufptr_t* buffer) {
void* ptr = &value;
return Codec<uint32_t>::encode(*reinterpret_cast<uint32_t*>(ptr), buffer);
}
};
template<typename T>
struct Codec<T, std::enable_if_t<std::is_enum<T>::value>> {
static std::optional<T> decode(cbufptr_t* buffer) {
std::optional<int32_t> int_val = SimpleSerializer<int32_t, false>::read(&(buffer->begin()), buffer->end());
return int_val.has_value() ? std::make_optional(static_cast<T>(int_val.value())) : std::nullopt;
}
static bool encode(T value, bufptr_t* buffer) { return SimpleSerializer<int32_t, false>::write(value, &(buffer->begin()), buffer->end()); }
};
template<> struct Codec<endpoint_ref_t> {
static std::optional<endpoint_ref_t> decode(cbufptr_t* buffer) {
std::optional<uint16_t> val0 = SimpleSerializer<uint16_t, false>::read(&(buffer->begin()), buffer->end());
std::optional<uint16_t> val1 = SimpleSerializer<uint16_t, false>::read(&(buffer->begin()), buffer->end());
return (val0.has_value() && val1.has_value()) ? std::make_optional(endpoint_ref_t{val1.value(), val0.value()}) : std::nullopt;
}
static bool encode(endpoint_ref_t value, bufptr_t* buffer) {
return SimpleSerializer<uint16_t, false>::write(value.endpoint_id, &(buffer->begin()), buffer->end())
&& SimpleSerializer<uint16_t, false>::write(value.json_crc, &(buffer->begin()), buffer->end());
}
};
}
/* @brief Handles the communication protocol on one channel.
*
* When instantiated with a list of endpoints and an output packet sink,
* objects of this class will handle packets passed into process_packet,
* pass the relevant data to the corresponding endpoints and dispatch response
* packets on the output.
*/
class BidirectionalPacketBasedChannel : public PacketSink {
public:
explicit BidirectionalPacketBasedChannel(PacketSink& output) :
output_(output)
{ }
//size_t get_mtu() {
// return SIZE_MAX;
//}
int process_packet(const uint8_t* buffer, size_t length) override;
private:
PacketSink& output_;
uint8_t tx_buf_[TX_BUF_SIZE] = {0};
};
/* ToString / FromString functions -------------------------------------------*/
/*
* These functions are currently not used by Fibre and only here to
* support the ODrive ASCII protocol.
* TODO: find a general way for client code to augment endpoints with custom
* functions
*/
template<typename T>
struct format_traits_t;
// template<> struct format_traits_t<float> { using type = void;
// static constexpr const char * fmt = "%f";
// static constexpr const char * fmtp = "%f";
// };
template<> struct format_traits_t<int64_t> { using type = void;
static constexpr const char * fmt = "%lld";
static constexpr const char * fmtp = "%lld";
};
template<> struct format_traits_t<uint64_t> { using type = void;
static constexpr const char * fmt = "%llu";
static constexpr const char * fmtp = "%llu";
};
template<> struct format_traits_t<int32_t> { using type = void;
static constexpr const char * fmt = "%ld";
static constexpr const char * fmtp = "%ld";
};
template<> struct format_traits_t<uint32_t> { using type = void;
static constexpr const char * fmt = "%lu";
static constexpr const char * fmtp = "%lu";
};
// TODO: change all overloads to fundamental int type space
template<> struct format_traits_t<unsigned int> { using type = void;
static constexpr const char * fmt = "%ud";
static constexpr const char * fmtp = "%ud";
};
template<> struct format_traits_t<int16_t> { using type = void;
static constexpr const char * fmt = "%hd";
static constexpr const char * fmtp = "%hd";
};
template<> struct format_traits_t<uint16_t> { using type = void;
static constexpr const char * fmt = "%hu";
static constexpr const char * fmtp = "%hu";
};
template<> struct format_traits_t<int8_t> { using type = void;
static constexpr const char * fmt = "%hhd";
static constexpr const char * fmtp = "%d";
};
template<> struct format_traits_t<uint8_t> { using type = void;
static constexpr const char * fmt = "%hhu";
static constexpr const char * fmtp = "%u";
};
template<typename T, typename = typename format_traits_t<T>::type>
static bool to_string(const T& value, char * buffer, size_t length, int) {
snprintf(buffer, length, format_traits_t<T>::fmtp, value);
return true;
}
// Special case for float because printf promotes float to double, and we get warnings
template<typename T = float>
static bool to_string(const float& value, char * buffer, size_t length, int) {
snprintf(buffer, length, "%f", (double)value);
return true;
}
template<typename T = bool>
static bool to_string(const bool& value, char * buffer, size_t length, int) {
buffer[0] = value ? '1' : '0';
buffer[1] = 0;
return true;
}
template<typename T>
static bool to_string(const T& value, char * buffer, size_t length, ...) {
return false;
}
template<typename T, typename = typename format_traits_t<T>::type>
static bool from_string(const char * buffer, size_t length, T* property, int) {
// Note for T == uint8_t: Even though we supposedly use the correct format
// string sscanf treats our pointer as pointer-to-int instead of
// pointer-to-uint8_t. To avoid an unexpected memory access we first read
// into a union.
union { T t; int i; } val;
if (sscanf(buffer, format_traits_t<T>::fmt, &val.t) == 1) {
*property = val.t;
return true;
} else {
return false;
}
}
// Special case for float because printf promotes float to double, and we get warnings
template<typename T = float>
static bool from_string(const char * buffer, size_t length, float* property, int) {
return sscanf(buffer, "%f", property) == 1;
}
template<typename T = bool>
static bool from_string(const char * buffer, size_t length, bool* property, int) {
int val;
if (sscanf(buffer, "%d", &val) != 1)
return false;
*property = val;
return true;
}
template<typename T>
static bool from_string(const char * buffer, size_t length, T* property, ...) {
return false;
}
//template<typename T, typename = typename std>
//bool set_from_float_ex(float value, T* property) {
// return false;
//}
namespace conversion {
//template<typename T>
template<typename T>
bool set_from_float_ex(float value, float* property, int) {
return *property = value, true;
}
template<typename T>
bool set_from_float_ex(float value, bool* property, int) {
return *property = (value >= 0.0f), true;
}
template<typename T, typename = std::enable_if_t<std::is_integral<T>::value && !std::is_const<T>::value>>
bool set_from_float_ex(float value, T* property, int) {
return *property = static_cast<T>(std::round(value)), true;
}
template<typename T>
bool set_from_float_ex(float value, T* property, ...) {
return false;
}
template<typename T>
bool set_from_float(float value, T* property) {
return set_from_float_ex<T>(value, property, 0);
}
}
template<typename T>
struct Property {
Property(void* ctx, T(*getter)(void*), void(*setter)(void*, T))
: ctx_(ctx), getter_(getter), setter_(setter) {}
Property(T* ctx)
: ctx_(ctx), getter_([](void* ctx){ return *(T*)ctx; }), setter_([](void* ctx, T val){ *(T*)ctx = val; }) {}
Property& operator*() { return *this; }
Property* operator->() { return this; }
T read() const {
return (*getter_)(ctx_);
}
T exchange(std::optional<T> value) const {
T old_value = (*getter_)(ctx_);
if (value.has_value()) {
(*setter_)(ctx_, value.value());
}
return old_value;
}
void* ctx_;
T(*getter_)(void*);
void(*setter_)(void*, T);
};
template<typename T>
struct Property<const T> {
Property(void* ctx, T(*getter)(void*))
: ctx_(ctx), getter_(getter) {}
Property(const T* ctx)
: ctx_(const_cast<T*>(ctx)), getter_([](void* ctx){ return *(const T*)ctx; }) {}
Property& operator*() { return *this; }
Property* operator->() { return this; }
T read() const {
return (*getter_)(ctx_);
}
void* ctx_;
T(*getter_)(void*);
};
#endif
@@ -0,0 +1,77 @@
#ifndef __FIBRE_SIMPLE_SERDES
#define __FIBRE_SIMPLE_SERDES
//#include "stream.hpp"
template<typename T, bool BigEndian, typename = void>
struct SimpleSerializer;
template<typename T>
using LittleEndianSerializer = SimpleSerializer<T, false>;
template<typename T>
using BigEndianSerializer = SimpleSerializer<T, true>;
/* @brief Serializer/deserializer for arbitrary integral number types */
// TODO: allow reading an arbitrary number of bits
template<typename T, bool BigEndian>
struct SimpleSerializer<T, BigEndian, typename std::enable_if_t<std::is_integral<T>::value>> {
static constexpr size_t BIT_WIDTH = std::numeric_limits<T>::digits;
static constexpr size_t BYTE_WIDTH = (BIT_WIDTH + 7) / 8;
template<typename TIterator>
static std::optional<T> read(TIterator* begin, TIterator end = nullptr) {
T result = 0;
if (BigEndian) {
for (size_t i = BYTE_WIDTH; i > 0; (i++, (*begin)++)) {
if (end && !(*begin < end))
return std::nullopt;
uint8_t byte = **begin;
result |= static_cast<T>(byte) << ((i - 1) << 3);
}
} else {
for (size_t i = 0; i < BYTE_WIDTH; (i++, (*begin)++)) {
if (end && !(*begin < end))
return std::nullopt;
uint8_t byte = **begin;
result |= static_cast<T>(byte) << (i << 3);
}
}
return result;
}
template<typename TIterator>
static bool write(T value, TIterator* begin, TIterator end = nullptr) {
if (BigEndian) {
for (size_t i = BYTE_WIDTH; i > 0; (i--, (*begin)++)) {
if (end && !(*begin < end))
return false;
uint8_t byte = static_cast<uint8_t>((value >> ((i - 1) << 3)) & 0xff);
**begin = byte;
}
} else {
for (size_t i = 0; i < BYTE_WIDTH; (i++, (*begin)++)) {
if (end && !(*begin < end))
return false;
uint8_t byte = static_cast<uint8_t>((value >> (i << 3)) & 0xff);
**begin = byte;
}
}
return true;
}
};
template<typename T>
inline std::optional<T> read_le(fibre::cbufptr_t* buffer) {
static_assert(is_complete<LittleEndianSerializer<T>>(), "no LittleEndianSerializer is defined for type T");
return LittleEndianSerializer<T>::read(&buffer->begin(), buffer->end());
}
template<typename T>
inline bool write_le(T value, fibre::bufptr_t* buffer) {
static_assert(is_complete<LittleEndianSerializer<T>>(), "no LittleEndianSerializer is defined for type T");
return LittleEndianSerializer<T>::write(value, &buffer->begin(), buffer->end());
}
#endif
@@ -0,0 +1,94 @@
/*[# This is the original template, thus the warning below does not apply to this file #]
* ============================ WARNING ============================
* ==== This is an autogenerated file. ====
* ==== Any changes to this file will be lost when recompiling. ====
* =================================================================
*
* This file contains base classes that correspond to the interfaces defined in
* your interface file. The objects you publish should inherit from these
* interfaces.
*
*/
#pragma GCC push_options
#pragma GCC optimize ("s")
[%- macro rettype(func) %]
[%- if not func.out -%]
void
[%- elif func.out | length == 1 -%]
[[(func.out.values() | first).type.c_name]]
[%- else -%]
[% for arg in func.out.values() %][[arg.type]][[', ' if not loop.last]][% endfor %]
[%- endif -%]
[%- endmacro %]
[%- macro render_interface(intf) %]
class [[intf.name | to_pascal_case]]Intf {
public:
[%- for intf in intf.interfaces -%]
[[render_interface(intf) | indent(4)]]
[%- endfor %]
[%- for enum in intf.enums %]
enum [[enum.name | to_pascal_case]] {
[%- for k, value in enum['values'].items() %]
[[((enum.name + k) | to_macro_case).ljust(32)]] = [% if enum.is_flags %]0x[['%08x' | format(value.value)]][% else %][[value.value]][% endif %],
[%- endfor %]
};
[%- endfor %]
[%- for property in intf.attributes.values() %]
[%- if property.type.fullname.startswith("fibre.Property") %]
[%- if not property.c_getter and not property.c_setter %]
template<typename T> static inline auto get_[[property.name]](T* obj) { return [[property.type.c_name]]{&obj->[[property.c_name]]}; }
template<typename T> static inline void get_[[property.name]](T* obj, void* ptr) { new (ptr) [[property.type.c_name]]{&obj->[[property.c_name]]}; }[# these are for the set_endpoint_from_float function. This is unmaintainable and should go away #]
[%- elif not property.c_setter %]
template<typename T> static inline auto get_[[property.name]](T* obj) { return [[property.type.c_name]]{obj, [](void* ctx){ return ([[property.type.value_type.c_name]])((T*)ctx)->[[property.c_getter]]; }}; }
template<typename T> static inline void get_[[property.name]](T* obj, void* ptr) { new (ptr) [[property.type.c_name]]{obj, [](void* ctx){ return ([[property.type.value_type.c_name]])((T*)ctx)->[[property.c_getter]]; }}; }
[%- else %]
template<typename T> static inline auto get_[[property.name]](T* obj) { return [[property.type.c_name]]{obj, [](void* ctx){ return ([[property.type.value_type.c_name]])((T*)ctx)->[[property.c_getter]]; }, [](void* ctx, [[property.type.value_type.c_name]] value){ ((T*)ctx)->[[property.c_setter]](value); }}; }
template<typename T> static inline void get_[[property.name]](T* obj, void* ptr) { new (ptr) [[property.type.c_name]]{obj, [](void* ctx){ return ([[property.type.value_type.c_name]])((T*)ctx)->[[property.c_getter]]; }, [](void* ctx, [[property.type.value_type.c_name]] value){ ((T*)ctx)->[[property.c_setter]](value); }}; }
[%- endif %]
[%- else %]
template<typename T> static inline auto get_[[property.name]](T* obj) { return &obj->[[property.c_name]]; }
[%- endif %]
[%- endfor %]
[%- for func in intf.functions.values() %]
virtual [[rettype(func)]] [[func.name | to_snake_case]]([% for in in func.in.values() %][% if loop.index0 %][[in.type.c_name]] [[in.name]][[', ' if not loop.last]][% endif %][% endfor %]) = 0;
[%- endfor %]
[%- for func in intf.functions.values() %]
[%- for k, arg in func.in.items() | skip_first %]
[[arg.type.c_name]] [[func.name | to_snake_case]]_in_[[arg.name]]_; // for internal use by Fibre
template<typename T> static auto get_[[func.name | to_snake_case]]_in_[[arg.name]]_(T* obj) { return Property<[[arg.type.c_name]]>{&obj->[[func.name | to_snake_case]]_in_[[arg.name]]_}; }
template<typename T> static void get_[[func.name | to_snake_case]]_in_[[arg.name]]_(T* obj, void* ptr) { new (ptr) Property<[[arg.type.c_name]]>{&obj->[[func.name | to_snake_case]]_in_[[arg.name]]_}; }
[%- endfor %]
[%- for k, arg in func.out.items() %]
[[arg.type.c_name]] [[func.name | to_snake_case]]_out_[[arg.name]]_; // for internal use by Fibre
template<typename T> static auto get_[[func.name | to_snake_case]]_out_[[arg.name]]_(T* obj) { return Property<const [[arg.type.c_name]]>{&obj->[[func.name | to_snake_case]]_out_[[arg.name]]_}; }
template<typename T> static void get_[[func.name | to_snake_case]]_out_[[arg.name]]_(T* obj, void* ptr) { new (ptr) Property<const [[arg.type.c_name]]>{&obj->[[func.name | to_snake_case]]_out_[[arg.name]]_}; }
[%- endfor %]
[%- endfor %]
};
[%- endmacro %]
[% for intf in toplevel_interfaces %]
[[render_interface(intf)]]
[% endfor %]
[%- for _, enum in value_types.items() %]
[%- if enum.is_flags %]
// this is technically not thread-safe but practically it might be
inline [[enum.c_name]] operator | ([[enum.c_name]] a, [[enum.c_name]] b) { return static_cast<[[enum.c_name]]>(static_cast<std::underlying_type_t<[[enum.c_name]]>>(a) | static_cast<std::underlying_type_t<[[enum.c_name]]>>(b)); }
inline [[enum.c_name]] operator & ([[enum.c_name]] a, [[enum.c_name]] b) { return static_cast<[[enum.c_name]]>(static_cast<std::underlying_type_t<[[enum.c_name]]>>(a) & static_cast<std::underlying_type_t<[[enum.c_name]]>>(b)); }
inline [[enum.c_name]] operator ^ ([[enum.c_name]] a, [[enum.c_name]] b) { return static_cast<[[enum.c_name]]>(static_cast<std::underlying_type_t<[[enum.c_name]]>>(a) ^ static_cast<std::underlying_type_t<[[enum.c_name]]>>(b)); }
inline [[enum.c_name]]& operator |= ([[enum.c_name]] &a, [[enum.c_name]] b) { return reinterpret_cast<[[enum.c_name]]&>(reinterpret_cast<std::underlying_type_t<[[enum.c_name]]>&>(a) |= static_cast<std::underlying_type_t<[[enum.c_name]]>>(b)); }
inline [[enum.c_name]]& operator &= ([[enum.c_name]] &a, [[enum.c_name]] b) { return reinterpret_cast<[[enum.c_name]]&>(reinterpret_cast<std::underlying_type_t<[[enum.c_name]]>&>(a) &= static_cast<std::underlying_type_t<[[enum.c_name]]>>(b)); }
inline [[enum.c_name]]& operator ^= ([[enum.c_name]] &a, [[enum.c_name]] b) { return reinterpret_cast<[[enum.c_name]]&>(reinterpret_cast<std::underlying_type_t<[[enum.c_name]]>&>(a) ^= static_cast<std::underlying_type_t<[[enum.c_name]]>>(b)); }
inline [[enum.c_name]] operator ~ ([[enum.c_name]] a) { return static_cast<[[enum.c_name]]>(~static_cast<std::underlying_type_t<[[enum.c_name]]>>(a)); }
[%- endif %]
[%- endfor %]
#pragma GCC pop_options
@@ -0,0 +1,8 @@
tup.include('../tupfiles/build.lua')
fibre_package = define_package{
sources={'protocol.cpp', 'posix_tcp.cpp', 'posix_udp.cpp'},
libs={'pthread'},
headers={'include'}
}
@@ -0,0 +1,108 @@
#include <arpa/inet.h>
#include <netinet/in.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <unistd.h>
#include <thread>
#include <future>
#include <vector>
#include <fibre/protocol.hpp>
#define TCP_RX_BUF_LEN 512
class TCPStreamSink : public StreamSink {
public:
TCPStreamSink(int socket_fd) :
socket_fd_(socket_fd)
{}
int process_bytes(const uint8_t* buffer, size_t length, size_t* processed_bytes) {
int bytes_sent = send(socket_fd_, buffer, length, 0);
if (processed_bytes)
*processed_bytes = (bytes_sent == -1) ? 0 : bytes_sent;
return (bytes_sent == -1) ? -1 : 0;
}
size_t get_free_space() { return SIZE_MAX; }
private:
int socket_fd_;
};
int serve_client(int sock_fd) {
uint8_t buf[TCP_RX_BUF_LEN];
// initialize output stack for this client
TCPStreamSink tcp_packet_output(sock_fd);
StreamBasedPacketSink packet2stream(tcp_packet_output);
BidirectionalPacketBasedChannel channel(packet2stream);
StreamToPacketSegmenter stream2packet(channel);
// now listen for it
for (;;) {
memset(buf, 0, sizeof(buf));
// returns as soon as there is some data
ssize_t n_received = recv(sock_fd, buf, sizeof(buf), 0);
// -1 indicates error and 0 means that the client gracefully terminated
if (n_received == -1 || n_received == 0) {
close(sock_fd);
return n_received;
}
// input processing stack
size_t processed = 0;
stream2packet.process_bytes(buf, n_received, &processed);
}
}
// function to check if a worker thread handling a single client is done
template<typename T>
bool future_is_ready(std::future<T>& t){
return t.wait_for(std::chrono::seconds(0)) == std::future_status::ready;
}
int serve_on_tcp(unsigned int port) {
struct sockaddr_in6 si_me, si_other;
int s;
if ((s=socket(AF_INET6, SOCK_STREAM, IPPROTO_TCP)) == -1) {
return -1;
}
memset((char *) &si_me, 0, sizeof(si_me));
si_me.sin6_family = AF_INET6;
si_me.sin6_port = htons(port);
si_me.sin6_flowinfo = 0;
si_me.sin6_addr = in6addr_any;
if (bind(s, reinterpret_cast<struct sockaddr *>(&si_me), sizeof(si_me)) == -1) {
return -1;
}
listen(s, 128); // make this socket a passive socket
std::vector<std::future<int>> serv_pool;
for (;;) {
memset(&si_other, 0, sizeof(si_other));
socklen_t silen = sizeof(si_other);
// TODO: Add a limit on accepting connections
int client_portal_fd = accept(s, reinterpret_cast<sockaddr *>(&si_other), &silen); // blocking call
serv_pool.push_back(std::async(std::launch::async, serve_client, client_portal_fd));
// do a little clean up on the pool
for (std::vector<std::future<int>>::iterator it = serv_pool.end()-1; it >= serv_pool.begin(); --it) {
if (future_is_ready(*it)) {
// we can erase this thread
serv_pool.erase(it);
}
}
}
close(s);
}
@@ -0,0 +1,70 @@
#include <arpa/inet.h>
#include <netinet/in.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <unistd.h>
#include <fibre/protocol.hpp>
#define UDP_RX_BUF_LEN 512
#define UDP_TX_BUF_LEN 512
class UDPPacketSender : public PacketSink {
public:
UDPPacketSender(int socket_fd, struct sockaddr_in6 *si_other) :
_socket_fd(socket_fd),
_si_other(si_other)
{}
size_t get_mtu() { return UDP_TX_BUF_LEN; }
int process_packet(const uint8_t* buffer, size_t length) {
// cannot send partial packets
if (length > get_mtu())
return -1;
int status = sendto(_socket_fd, buffer, length, 0, reinterpret_cast<struct sockaddr*>(_si_other), sizeof(*_si_other));
return (status == -1) ? -1 : 0;
}
private:
int _socket_fd;
struct sockaddr_in6 *_si_other;
};
int serve_on_udp(unsigned int port) {
struct sockaddr_in6 si_me, si_other;
int s;
socklen_t slen = sizeof(si_other);
uint8_t buf[UDP_RX_BUF_LEN];
if ((s=socket(AF_INET6, SOCK_DGRAM, IPPROTO_UDP)) == -1)
return -1;
memset((char *) &si_me, 0, sizeof(si_me));
si_me.sin6_family = AF_INET6;
si_me.sin6_port = htons(port);
si_me.sin6_flowinfo = 0;
si_me.sin6_addr= in6addr_any;
if (bind(s, reinterpret_cast<struct sockaddr *>(&si_me), sizeof(si_me)) == -1)
return -1;
for (;;) {
ssize_t n_received = recvfrom(s, buf, sizeof(buf), 0, reinterpret_cast<struct sockaddr *>(&si_other), &slen);
if (n_received == -1)
return -1;
//printf("Received packet from %s:%d\nData: %s\n\n",
// inet_ntoa(si_other.sin_addr), ntohs(si_other.sin_port), buf);
UDPPacketSender udp_packet_output(s, &si_other);
BidirectionalPacketBasedChannel udp_channel(udp_packet_output);
udp_channel.process_packet(buf, n_received);
}
close(s);
}
@@ -0,0 +1,187 @@
/* Includes ------------------------------------------------------------------*/
#include <memory>
#include <stdlib.h>
#include <fibre/protocol.hpp>
#include <fibre/crc.hpp>
/* Private defines -----------------------------------------------------------*/
/* Private macros ------------------------------------------------------------*/
/* Private typedef -----------------------------------------------------------*/
/* Global constant data ------------------------------------------------------*/
/* Global variables ----------------------------------------------------------*/
/* Private constant data -----------------------------------------------------*/
/* Private variables ---------------------------------------------------------*/
/* Private function prototypes -----------------------------------------------*/
static void hexdump(const uint8_t* buf, size_t len);
/* Function implementations --------------------------------------------------*/
#if 0
void hexdump(const uint8_t* buf, size_t len) {
for (size_t pos = 0; pos < len; ++pos) {
printf(" %02x", buf[pos]);
if ((((pos + 1) % 16) == 0) || ((pos + 1) == len))
printf("\r\n");
osDelay(2);
}
}
#else
void hexdump(const uint8_t* buf, size_t len) {
(void) buf;
(void) len;
}
#endif
int StreamToPacketSegmenter::process_bytes(const uint8_t *buffer, size_t length, size_t* processed_bytes) {
int result = 0;
while (length--) {
if (header_index_ < sizeof(header_buffer_)) {
// Process header byte
header_buffer_[header_index_++] = *buffer;
if (header_index_ == 1 && header_buffer_[0] != CANONICAL_PREFIX) {
header_index_ = 0;
} else if (header_index_ == 2 && (header_buffer_[1] & 0x80)) {
header_index_ = 0; // TODO: support packets larger than 128 bytes
} else if (header_index_ == 3 && calc_crc8<CANONICAL_CRC8_POLYNOMIAL>(CANONICAL_CRC8_INIT, header_buffer_, 3)) {
header_index_ = 0;
} else if (header_index_ == 3) {
packet_length_ = header_buffer_[1] + 2;
}
} else if (packet_index_ < sizeof(packet_buffer_)) {
// Process payload byte
packet_buffer_[packet_index_++] = *buffer;
}
// If both header and packet are fully received, hand it on to the packet processor
if (header_index_ == 3 && packet_index_ == packet_length_) {
if (calc_crc16<CANONICAL_CRC16_POLYNOMIAL>(CANONICAL_CRC16_INIT, packet_buffer_, packet_length_) == 0) {
result |= output_.process_packet(packet_buffer_, packet_length_ - 2);
}
header_index_ = packet_index_ = packet_length_ = 0;
}
buffer++;
if (processed_bytes)
(*processed_bytes)++;
}
return result;
}
int StreamBasedPacketSink::process_packet(const uint8_t *buffer, size_t length) {
// TODO: support buffer size >= 128
if (length >= 128)
return -1;
LOG_FIBRE("send header\r\n");
uint8_t header[] = {
CANONICAL_PREFIX,
static_cast<uint8_t>(length),
0
};
header[2] = calc_crc8<CANONICAL_CRC8_POLYNOMIAL>(CANONICAL_CRC8_INIT, header, 2);
if (output_.process_bytes(header, sizeof(header), nullptr))
return -1;
LOG_FIBRE("send payload:\r\n");
hexdump(buffer, length);
if (output_.process_bytes(buffer, length, nullptr))
return -1;
LOG_FIBRE("send crc16\r\n");
uint16_t crc16 = calc_crc16<CANONICAL_CRC16_POLYNOMIAL>(CANONICAL_CRC16_INIT, buffer, length);
uint8_t crc16_buffer[] = {
(uint8_t)((crc16 >> 8) & 0xff),
(uint8_t)((crc16 >> 0) & 0xff)
};
if (output_.process_bytes(crc16_buffer, 2, nullptr))
return -1;
LOG_FIBRE("sent!\r\n");
return 0;
}
// Returns part of the JSON interface definition.
bool fibre::endpoint0_handler(fibre::cbufptr_t* input_buffer, fibre::bufptr_t* output_buffer) {
// The request must contain a 32 bit integer to specify an offset
std::optional<uint32_t> offset = read_le<uint32_t>(input_buffer);
if (!offset.has_value()) {
// Didn't receive any offset
return false;
} else if (offset.value() == 0xffffffff) {
// If the offset is special value 0xFFFFFFFF, send back the JSON version ID instead
return write_le<uint32_t>(json_version_id_, output_buffer);
} else if (offset.value() >= embedded_json_length) {
// Attempt to read beyond the buffer end - return empty response
return true;
} else {
// Return part of the json file
size_t n_copy = std::min(output_buffer->size(), embedded_json_length - (size_t)offset.value());
memcpy(output_buffer->begin(), embedded_json + offset.value(), n_copy);
*output_buffer = output_buffer->skip(n_copy);
return true;
}
}
int BidirectionalPacketBasedChannel::process_packet(const uint8_t* buffer, size_t length) {
LOG_FIBRE("got packet of length %d: \r\n", length);
hexdump(buffer, length);
if (length < 4)
return -1;
uint16_t seq_no = read_le<uint16_t>(&buffer, &length);
if (seq_no & 0x8000) {
// TODO: ack handling
} else {
// TODO: think about some kind of ordering guarantees
// currently the seq_no is just used to associate a response with a request
uint16_t endpoint_id = read_le<uint16_t>(&buffer, &length);
bool expect_response = endpoint_id & 0x8000;
endpoint_id &= 0x7fff;
// Verify packet trailer. The expected trailer value depends on the selected endpoint.
// For endpoint 0 this is just the protocol version, for all other endpoints it's a
// CRC over the entire JSON descriptor tree (this may change in future versions).
uint16_t expected_trailer = endpoint_id ? fibre::json_crc_ : PROTOCOL_VERSION;
uint16_t actual_trailer = buffer[length - 2] | (buffer[length - 1] << 8);
if (expected_trailer != actual_trailer) {
LOG_FIBRE("trailer mismatch for endpoint %d: expected %04x, got %04x\r\n", endpoint_id, expected_trailer, actual_trailer);
return -1;
}
LOG_FIBRE("trailer ok for endpoint %d\r\n", endpoint_id);
// TODO: if more bytes than the MTU were requested, should we abort or just return as much as possible?
uint16_t expected_response_length = read_le<uint16_t>(&buffer, &length);
// Limit response length according to our local TX buffer size
if (expected_response_length > sizeof(tx_buf_) - 2)
expected_response_length = sizeof(tx_buf_) - 2;
fibre::cbufptr_t input_buffer{buffer, length - 2};
fibre::bufptr_t output_buffer{tx_buf_ + 2, expected_response_length};
fibre::endpoint_handler(endpoint_id, &input_buffer, &output_buffer);
// Send response
if (expect_response) {
size_t actual_response_length = expected_response_length - output_buffer.size() + 2;
write_le<uint16_t>(seq_no | 0x8000, tx_buf_);
LOG_FIBRE("send packet:\r\n");
hexdump(tx_buf_, actual_response_length);
output_.process_packet(tx_buf_, actual_response_length);
}
}
return 0;
}
@@ -0,0 +1,50 @@
/*[# This is the original template, thus the warning below does not apply to this file #]
* ============================ WARNING ============================
* ==== This is an autogenerated file. ====
* ==== Any changes to this file will be lost when recompiling. ====
* =================================================================
*
* This file contains support functions for the ODrive ASCII protocol.
*
* TODO: might generalize this as an approach to runtime introspection.
*/
#include <fibre/introspection.hpp>
#pragma GCC push_options
#pragma GCC optimize ("s")
[% for intf in interfaces.values() %][% if not intf.builtin %]
template<typename T>
struct [[intf.fullname | to_pascal_case]]TypeInfo : TypeInfo {
using TypeInfo::TypeInfo;
static const PropertyInfo property_table[];
static const [[intf.fullname | to_pascal_case]]TypeInfo<T> singleton;
static Introspectable make_introspectable(T& obj) { return TypeInfo::make_introspectable(&obj, &singleton); }
introspectable_storage_t get_child(introspectable_storage_t obj, size_t idx) const override {
T* ptr = *(T**)&obj;
introspectable_storage_t res;
switch (idx) {
[%- for property in intf.attributes.values() %]
case [[loop.index0]]: *(decltype([[intf.c_name]]::get_[[property.name]](std::declval<T*>()))*)(&res) = [[intf.c_name]]::get_[[property.name]](ptr); break;
[%- endfor %]
}
return res;
}
};
[% endif %][% endfor %]
[% for intf in interfaces.values() %][% if not intf.builtin %]
template<typename T>
const PropertyInfo [[intf.fullname | to_pascal_case]]TypeInfo<T>::property_table[] = {
[%- for property in intf.attributes.values() %]
{"[[property.name]]", &[[(property.type.purename or property.type.fullname) | to_pascal_case]]TypeInfo<std::remove_reference_t<decltype(*[[intf.c_name]]::get_[[property.name]](std::declval<T*>()))>>::singleton},
[%- endfor %]
};
template<typename T>
const [[intf.fullname | to_pascal_case]]TypeInfo<T> [[intf.fullname | to_pascal_case]]TypeInfo<T>::singleton{[[intf.fullname | to_pascal_case]]TypeInfo<T>::property_table, sizeof([[intf.fullname | to_pascal_case]]TypeInfo<T>::property_table) / sizeof([[intf.fullname | to_pascal_case]]TypeInfo<T>::property_table[0])};
[% endif %][% endfor %]
#pragma GCC pop_options
@@ -0,0 +1,16 @@
# Python Distribution / packaging
.Python
/dist/
/*.egg-info/
/MANIFEST
# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec
# Installer logs
pip-log.txt
pip-delete-this-directory.txt
@@ -0,0 +1,5 @@
from .discovery import find_any, find_all
from .utils import Event, Logger, TimeoutError
from .protocol import ChannelBrokenException, ChannelDamagedException
from .shell import launch_shell
@@ -0,0 +1,179 @@
"""
Provides functions for the discovery of Fibre nodes
"""
import sys
import json
import time
import threading
import traceback
import struct
import fibre.protocol
import fibre.utils
import fibre.remote_object
from fibre.utils import Event, Logger
from fibre.protocol import ChannelBrokenException, TimeoutError
import appdirs
import os
# Load all installed transport layers
channel_types = {}
try:
import fibre.usbbulk_transport
channel_types['usb'] = fibre.usbbulk_transport.discover_channels
except ImportError:
pass
try:
import fibre.serial_transport
channel_types['serial'] = fibre.serial_transport.discover_channels
except ImportError:
pass
try:
import fibre.tcp_transport
channel_types['tcp'] = fibre.tcp_transport.discover_channels
except ImportError:
pass
try:
import fibre.udp_transport
channel_types['udp'] = fibre.udp_transport.discover_channels
except ImportError:
pass
def noprint(text):
pass
def find_all(path, serial_number,
did_discover_object_callback,
search_cancellation_token,
channel_termination_token,
logger):
"""
Starts scanning for Fibre nodes that match the specified path spec and calls
the callback for each Fibre node that is found.
This function is non-blocking.
"""
def did_discover_channel(channel):
"""
Inits an object from a given channel and then calls did_discover_object_callback
with the created object
This queries the endpoint 0 on that channel to gain information
about the interface, which is then used to init the corresponding object.
"""
try:
logger.debug("Connecting to device on " + channel._name)
cache_dir = appdirs.user_cache_dir("odrivetool")
cache_path = None
# Fetch the json version tag to check cache (only supported on firmware v0.5 or later)
try:
json_version_tag = channel.remote_endpoint_operation(0, struct.pack("<I", 0xffffffff), True, 4)
json_version_tag = struct.unpack("<I", json_version_tag)[0]
logger.debug("Device reported JSON version ID: {:08d}".format(json_version_tag))
cache_path = os.path.join(cache_dir, 'fibre_schema_cache_{:08d}'.format(json_version_tag))
except:
logger.debug("Failed to get JSON checksum")
# Check cache
json_data = None
try:
if not cache_path is None:
with open(cache_path, 'rb') as fp:
json_crc16 = fibre.protocol.calc_crc16(fibre.protocol.PROTOCOL_VERSION, fp.read())
fp.seek(0)
json_data = json.load(fp)
except:
logger.debug("Failed load JSON cache file {}".format(cache_path))
# Fallback to loading JSON from device
if json_data is None:
# Downloading json data
logger.info("Downloading json data from ODrive... (this might take a while)")
json_bytes = channel.remote_endpoint_read_buffer(0)
try:
json_string = json_bytes.decode("ascii")
except UnicodeDecodeError:
logger.debug("Device responded on endpoint 0 with something that is not ASCII")
raise UnicodeDecodeError
json_crc16 = fibre.protocol.calc_crc16(fibre.protocol.PROTOCOL_VERSION, json_bytes)
json_data = json.loads(json_string)
# Save JSON to cache
if not cache_path is None:
logger.debug("Creating new JSON cache file {}".format(cache_path))
os.makedirs(cache_dir, exist_ok=True)
with open(cache_path, 'w+') as json_cache:
json_cache.write(json_string)
logger.debug("Saved JSON to cache file {}".format(cache_path))
channel._interface_definition_crc = json_crc16
logger.debug("JSON: " + str(json_data).replace("{'name'", "\n{'name'"))
json_data = {"name": "fibre_node", "members": json_data}
obj = fibre.remote_object.RemoteObject(json_data, None, channel, logger)
obj.__dict__['_json_data'] = json_data['members']
obj.__dict__['_json_crc'] = json_crc16
device_serial_number = fibre.utils.get_serial_number_str(obj)
if serial_number != None and device_serial_number != serial_number:
logger.debug("Ignoring device with serial number {}".format(device_serial_number))
return
did_discover_object_callback(obj)
except Exception:
logger.debug("Unexpected exception after discovering channel: " + traceback.format_exc())
# For each connection type, kick off an appropriate discovery loop
for search_spec in path.split(','):
prefix = search_spec.split(':')[0]
the_rest = ':'.join(search_spec.split(':')[1:])
if prefix in channel_types:
t = threading.Thread(target=channel_types[prefix],
args=(the_rest, serial_number, did_discover_channel, search_cancellation_token, channel_termination_token, logger))
t.daemon = True
t.start()
else:
raise Exception("Invalid path spec \"{}\"".format(search_spec))
def find_any(path="usb", serial_number=None,
search_cancellation_token=None, channel_termination_token=None,
timeout=None, logger=Logger(verbose=False), find_multiple=False):
"""
Blocks until the first matching Fibre node is connected and then returns that node
"""
result = []
done_signal = Event(search_cancellation_token)
def did_discover_object(obj):
result.append(obj)
if find_multiple:
if len(result) >= int(find_multiple):
done_signal.set()
else:
done_signal.set()
find_all(path, serial_number, did_discover_object, done_signal, channel_termination_token, logger)
try:
done_signal.wait(timeout=timeout)
except TimeoutError:
if not find_multiple:
return None
finally:
done_signal.set() # terminate find_all
if find_multiple:
return result
else:
return result[0]
@@ -0,0 +1,359 @@
# See protocol.hpp for an overview of the protocol
import time
import struct
import sys
import threading
import traceback
#import fibre.utils
from fibre.utils import Event, wait_any, TimeoutError
import abc
if sys.version_info >= (3, 4):
ABC = abc.ABC
else:
ABC = abc.ABCMeta('ABC', (), {})
if sys.version_info < (3, 3):
from monotonic import monotonic
time.monotonic = monotonic
SYNC_BYTE = 0xAA
CRC8_INIT = 0x42
CRC16_INIT = 0x1337
PROTOCOL_VERSION = 1
CRC8_DEFAULT = 0x37 # this must match the polynomial in the C++ implementation
CRC16_DEFAULT = 0x3d65 # this must match the polynomial in the C++ implementation
MAX_PACKET_SIZE = 128
# For more information on the CRC algorithm refer to protocol.md
def calc_crc(remainder, value, polynomial, bitwidth):
topbit = (1 << (bitwidth - 1))
# Bring the next byte into the remainder.
remainder ^= (value << (bitwidth - 8))
for bitnumber in range(0,8):
if (remainder & topbit):
remainder = (remainder << 1) ^ polynomial
else:
remainder = (remainder << 1)
return remainder & ((1 << bitwidth) - 1)
def calc_crc8(remainder, value):
if isinstance(value, bytearray) or isinstance(value, bytes) or isinstance(value, list):
for byte in value:
if not isinstance(byte,int):
byte = ord(byte)
remainder = calc_crc(remainder, byte, CRC8_DEFAULT, 8)
else:
remainder = calc_crc(remainder, byte, CRC8_DEFAULT, 8)
return remainder
def calc_crc16(remainder, value):
if isinstance(value, bytearray) or isinstance(value, bytes) or isinstance(value, list):
for byte in value:
if not isinstance(byte, int):
byte = ord(byte)
remainder = calc_crc(remainder, byte, CRC16_DEFAULT, 16)
else:
remainder = calc_crc(remainder, value, CRC16_DEFAULT, 16)
return remainder
class DeviceInitException(Exception):
pass
class ChannelDamagedException(Exception):
"""
Raised when the channel is temporarily broken and a
resend of the message might be successful
"""
pass
class ChannelBrokenException(Exception):
"""
Raised when the channel is permanently broken
"""
pass
class StreamSource(ABC):
@abc.abstractmethod
def get_bytes(self, n_bytes, deadline):
pass
class StreamSink(ABC):
@abc.abstractmethod
def process_bytes(self, bytes):
pass
class PacketSource(ABC):
@abc.abstractmethod
def get_packet(self, deadline):
pass
class PacketSink(ABC):
@abc.abstractmethod
def process_packet(self, packet):
pass
class StreamToPacketSegmenter(StreamSink):
def __init__(self, output):
self._header = []
self._packet = []
self._packet_length = 0
self._output = output
def process_bytes(self, bytes):
"""
Processes an arbitrary number of bytes. If one or more full packets are
are received, they are sent to this instance's output PacketSink.
Incomplete packets are buffered between subsequent calls to this function.
"""
for byte in bytes:
if (len(self._header) < 3):
# Process header byte
self._header.append(byte)
if (len(self._header) == 1) and (self._header[0] != SYNC_BYTE):
self._header = []
elif (len(self._header) == 2) and (self._header[1] & 0x80):
self._header = [] # TODO: support packets larger than 128 bytes
elif (len(self._header) == 3) and calc_crc8(CRC8_INIT, self._header):
self._header = []
elif (len(self._header) == 3):
self._packet_length = self._header[1] + 2
else:
# Process payload byte
self._packet.append(byte)
# If both header and packet are fully received, hand it on to the packet processor
if (len(self._header) == 3) and (len(self._packet) == self._packet_length):
if calc_crc16(CRC16_INIT, self._packet) == 0:
self._output.process_packet(self._packet[:-2])
self._header = []
self._packet = []
self._packet_length = 0
class StreamBasedPacketSink(PacketSink):
def __init__(self, output):
self._output = output
def process_packet(self, packet):
if (len(packet) >= MAX_PACKET_SIZE):
raise NotImplementedError("packet larger than 127 currently not supported")
header = bytearray()
header.append(SYNC_BYTE)
header.append(len(packet))
header.append(calc_crc8(CRC8_INIT, header))
self._output.process_bytes(header)
self._output.process_bytes(packet)
# append CRC in big endian
crc16 = calc_crc16(CRC16_INIT, packet)
self._output.process_bytes(struct.pack('>H', crc16))
class PacketFromStreamConverter(PacketSource):
def __init__(self, input):
self._input = input
def get_packet(self, deadline):
"""
Requests bytes from the underlying input stream until a full packet is
received or the deadline is reached, in which case None is returned. A
deadline before the current time corresponds to non-blocking mode.
"""
while True:
header = bytes()
# TODO: sometimes this call hangs, even though the device apparently sent something
header = header + self._input.get_bytes_or_fail(1, deadline)
if (header[0] != SYNC_BYTE):
#print("sync byte mismatch")
continue
header = header + self._input.get_bytes_or_fail(1, deadline)
if (header[1] & 0x80):
#print("packet too large")
continue # TODO: support packets larger than 128 bytes
header = header + self._input.get_bytes_or_fail(1, deadline)
if calc_crc8(CRC8_INIT, header) != 0:
#print("crc8 mismatch")
continue
packet_length = header[1] + 2
#print("wait for {} bytes".format(packet_length))
packet = self._input.get_bytes_or_fail(packet_length, deadline)
if calc_crc16(CRC16_INIT, packet) != 0:
#print("crc16 mismatch")
continue
return packet[:-2]
class Channel(PacketSink):
# Choose these parameters to be sensible for a specific transport layer
_resend_timeout = 5.0 # [s]
_send_attempts = 5
def __init__(self, name, input, output, cancellation_token, logger):
"""
Params:
input: A PacketSource where this channel will source packets from on
demand. Alternatively packets can be provided to this channel
directly by calling process_packet on this instance.
output: A PacketSink where this channel will put outgoing packets.
"""
self._name = name
self._input = input
self._output = output
self._logger = logger
self._outbound_seq_no = 0
self._interface_definition_crc = 0
self._expected_acks = {}
self._responses = {}
self._my_lock = threading.Lock()
self._channel_broken = Event(cancellation_token)
self.start_receiver_thread(Event(self._channel_broken))
def start_receiver_thread(self, cancellation_token):
"""
Starts the receiver thread that processes incoming messages.
The thread quits as soon as the channel enters a broken state.
"""
def receiver_thread():
error_ctr = 0
try:
while (not cancellation_token.is_set() and not self._channel_broken.is_set()
and error_ctr < 10):
# Set an arbitrary deadline because the get_packet function
# currently doesn't support a cancellation_token
deadline = time.monotonic() + 1.0
try:
response = self._input.get_packet(deadline)
except TimeoutError:
continue # try again
except ChannelDamagedException:
error_ctr += 1
continue # try again
if (error_ctr > 0):
error_ctr -= 1
# Process response
# This should not throw an exception, otherwise the channel breaks
self.process_packet(response)
#print("receiver thread is exiting")
except Exception:
self._logger.debug("receiver thread is exiting: " + traceback.format_exc())
finally:
self._channel_broken.set()
t = threading.Thread(target=receiver_thread)
t.daemon = True
t.start()
def remote_endpoint_operation(self, endpoint_id, input, expect_ack, output_length):
if input is None:
input = bytearray(0)
if (len(input) >= 128):
raise Exception("packet larger than 127 currently not supported")
if (expect_ack):
endpoint_id |= 0x8000
self._my_lock.acquire()
try:
self._outbound_seq_no = ((self._outbound_seq_no + 1) & 0x7fff)
seq_no = self._outbound_seq_no
finally:
self._my_lock.release()
seq_no |= 0x80 # FIXME: we hardwire one bit of the seq-no to 1 to avoid conflicts with the ascii protocol
packet = struct.pack('<HHH', seq_no, endpoint_id, output_length)
packet = packet + input
crc16 = calc_crc16(CRC16_INIT, packet)
if (endpoint_id & 0x7fff == 0):
trailer = PROTOCOL_VERSION
else:
trailer = self._interface_definition_crc
#print("append trailer " + trailer)
packet = packet + struct.pack('<H', trailer)
if (expect_ack):
ack_event = Event()
self._expected_acks[seq_no] = ack_event
try:
attempt = 0
while (attempt < self._send_attempts):
self._my_lock.acquire()
try:
self._output.process_packet(packet)
except ChannelDamagedException:
attempt += 1
continue # resend
except TimeoutError:
attempt += 1
continue # resend
finally:
self._my_lock.release()
# Wait for ACK until the resend timeout is exceeded
try:
if wait_any(self._resend_timeout, ack_event, self._channel_broken) != 0:
raise ChannelBrokenException()
except TimeoutError:
attempt += 1
continue # resend
return self._responses.pop(seq_no)
# TODO: record channel statistics
raise ChannelBrokenException() # Too many resend attempts
finally:
self._expected_acks.pop(seq_no)
self._responses.pop(seq_no, None)
else:
# fire and forget
self._output.process_packet(packet)
return None
def remote_endpoint_read_buffer(self, endpoint_id):
"""
Handles reads from long endpoints
"""
# TODO: handle device that could (maliciously) send infinite stream
buffer = bytes()
while True:
chunk_length = 512
chunk = self.remote_endpoint_operation(endpoint_id, struct.pack("<I", len(buffer)), True, chunk_length)
if (len(chunk) == 0):
break
buffer += chunk
return buffer
def process_packet(self, packet):
#print("process packet")
packet = bytes(packet)
if (len(packet) < 2):
raise Exception("packet too short")
seq_no = struct.unpack('<H', packet[0:2])[0]
if (seq_no & 0x8000):
seq_no &= 0x7fff
ack_signal = self._expected_acks.get(seq_no, None)
if (ack_signal):
self._responses[seq_no] = packet[2:]
ack_signal.set()
#print("received ack for packet " + str(seq_no))
else:
print("received unexpected ACK: " + str(seq_no))
else:
#if (calc_crc16(CRC16_INIT, struct.pack('<HBB', PROTOCOL_VERSION, packet[-2], packet[-1]))):
# raise Exception("CRC16 mismatch")
print("endpoint requested")
# TODO: handle local endpoint operation
@@ -0,0 +1,264 @@
"""
Provides functions for the discovery of Fibre nodes
"""
import sys
import json
import struct
import threading
import fibre.protocol
class ObjectDefinitionError(Exception):
pass
codecs = {}
class StructCodec():
"""
Generic serializer/deserializer based on struct pack
"""
def __init__(self, struct_format, target_type):
self._struct_format = struct_format
self._target_type = target_type
def get_length(self):
return struct.calcsize(self._struct_format)
def serialize(self, value):
value = self._target_type(value)
return struct.pack(self._struct_format, value)
def deserialize(self, buffer):
value = struct.unpack(self._struct_format, buffer)
value = value[0] if len(value) == 1 else value
return self._target_type(value)
class RemoteProperty():
"""
Used internally by dynamically created objects to translate
property assignments and fetches into endpoint operations on the
object's associated channel
"""
def __init__(self, json_data, parent):
self._parent = parent
self.__channel__ = parent.__channel__
id_str = json_data.get("id", None)
if id_str is None:
raise ObjectDefinitionError("unspecified endpoint ID")
self._id = int(id_str)
self._name = json_data.get("name", None)
if self._name is None:
self._name = "[anonymous]"
type_str = json_data.get("type", None)
if type_str is None:
raise ObjectDefinitionError("unspecified type")
# Find all codecs that match the type_str and build a dictionary
# of the form {type1: codec1, type2: codec2}
eligible_types = {k: v[type_str] for (k,v) in codecs.items() if type_str in v}
if not eligible_types:
raise ObjectDefinitionError("unsupported codec {}".format(type_str))
# TODO: better heuristics to select a matching type (i.e. prefer non lossless)
eligible_types = list(eligible_types.items())
self._property_type = eligible_types[0][0]
self._codec = eligible_types[0][1]
access_mode = json_data.get("access", "r")
self._can_read = 'r' in access_mode
self._can_write = 'w' in access_mode
def get_value(self):
buffer = self._parent.__channel__.remote_endpoint_operation(self._id, None, True, self._codec.get_length())
return self._codec.deserialize(buffer)
def set_value(self, value):
buffer = self._codec.serialize(value)
# TODO: Currenly we wait for an ack here. Settle on the default guarantee.
self._parent.__channel__.remote_endpoint_operation(self._id, buffer, True, 0)
def _dump(self):
if self._name == "serial_number":
# special case: serial number should be displayed in hex (TODO: generalize)
val_str = "{:012X}".format(self.get_value())
elif self._name == "error":
# special case: errors should be displayed in hex (TODO: generalize)
val_str = "0x{:04X}".format(self.get_value())
else:
val_str = str(self.get_value())
return "{} = {} ({})".format(self._name, val_str, self._property_type.__name__)
class EndpointRefCodec():
"""
Serializer/deserializer for an endpoint reference
"""
def get_length(self):
return struct.calcsize("<HH")
def serialize(self, value):
if value is None:
(ep_id, ep_crc) = (0, 0)
elif isinstance(value, RemoteProperty):
(ep_id, ep_crc) = (value._id, value.__channel__._interface_definition_crc)
else:
raise TypeError("Expected value of type RemoteProperty or None but got '{}'. En example for a RemoteProperty is this expression: odrv0.axis0.controller._remote_attributes['pos_setpoint']".format(type(value).__name__))
return struct.pack("<HH", ep_id, ep_crc)
def deserialize(self, buffer):
return struct.unpack("<HH", buffer)
codecs[int] = {
'int8': StructCodec("<b", int),
'uint8': StructCodec("<B", int),
'int16': StructCodec("<h", int),
'uint16': StructCodec("<H", int),
'int32': StructCodec("<i", int),
'uint32': StructCodec("<I", int),
'int64': StructCodec("<q", int),
'uint64': StructCodec("<Q", int)
}
codecs[bool] = {
'bool': StructCodec("<?", bool)
}
codecs[float] = {
'float': StructCodec("<f", float)
}
codecs[RemoteProperty] = {
'endpoint_ref': EndpointRefCodec()
}
class RemoteFunction(object):
"""
Represents a callable function that maps to a function call on a remote object
"""
def __init__(self, json_data, parent):
self._parent = parent
id_str = json_data.get("id", None)
if id_str is None:
raise ObjectDefinitionError("unspecified endpoint ID")
self._trigger_id = int(id_str)
self._name = json_data.get("name", None)
if self._name is None:
self._name = "[anonymous]"
self._inputs = []
for param_json in json_data.get("arguments", []) + json_data.get("inputs", []): # TODO: deprecate "arguments" keyword
param_json["mode"] = "r"
self._inputs.append(RemoteProperty(param_json, parent))
self._outputs = []
for param_json in json_data.get("outputs", []): # TODO: deprecate "arguments" keyword
param_json["mode"] = "r"
self._outputs.append(RemoteProperty(param_json, parent))
def __call__(self, *args):
if (len(self._inputs) != len(args)):
raise TypeError("expected {} arguments but have {}".format(len(self._inputs), len(args)))
for i in range(len(args)):
self._inputs[i].set_value(args[i])
self._parent.__channel__.remote_endpoint_operation(self._trigger_id, None, True, 0)
if len(self._outputs) > 0:
return self._outputs[0].get_value()
def _dump(self):
return "{}({})".format(self._name, ", ".join("{}: {}".format(x._name, x._property_type.__name__) for x in self._inputs))
class RemoteObject(object):
"""
Object with functions and properties that map to remote endpoints
"""
def __init__(self, json_data, parent, channel, logger):
"""
Creates an object that implements the specified JSON type description by
communicating over the provided channel
"""
# Directly write to __dict__ to avoid calling __setattr__ too early
object.__getattribute__(self, "__dict__")["_remote_attributes"] = {}
object.__getattribute__(self, "__dict__")["__sealed__"] = False
# Assign once more to make linter happy
self._remote_attributes = {}
self.__sealed__ = False
self.__channel__ = channel
self.__parent__ = parent
# Build attribute list from JSON
for member_json in json_data.get("members", []):
member_name = member_json.get("name", None)
if member_name is None:
logger.debug("ignoring unnamed attribute")
continue
try:
type_str = member_json.get("type", None)
if type_str == "object":
attribute = RemoteObject(member_json, self, channel, logger)
elif type_str == "function":
attribute = RemoteFunction(member_json, self)
elif type_str != None:
attribute = RemoteProperty(member_json, self)
else:
raise ObjectDefinitionError("no type information")
except ObjectDefinitionError as ex:
logger.debug("malformed member {}: {}".format(member_name, str(ex)))
continue
self._remote_attributes[member_name] = attribute
self.__dict__[member_name] = attribute
# Ensure that from here on out assignments to undefined attributes
# raise an exception
self.__sealed__ = True
channel._channel_broken.subscribe(self._tear_down)
def _dump(self, indent, depth):
if depth <= 0:
return "..."
lines = []
for key, val in self._remote_attributes.items():
if isinstance(val, RemoteObject):
val_str = indent + key + (": " if depth == 1 else ":\n") + val._dump(indent + " ", depth - 1)
else:
val_str = indent + val._dump()
lines.append(val_str)
return "\n".join(lines)
def __str__(self):
return self._dump("", depth=2)
def __repr__(self):
return self.__str__()
def __getattribute__(self, name):
attr = object.__getattribute__(self, "_remote_attributes").get(name, None)
if isinstance(attr, RemoteProperty):
if attr._can_read:
return attr.get_value()
else:
raise Exception("Cannot read from property {}".format(name))
elif attr != None:
return attr
else:
return object.__getattribute__(self, name)
#raise AttributeError("Attribute {} not found".format(name))
def __setattr__(self, name, value):
attr = object.__getattribute__(self, "_remote_attributes").get(name, None)
if isinstance(attr, RemoteProperty):
if attr._can_write:
attr.set_value(value)
else:
raise Exception("Cannot write to property {}".format(name))
elif not object.__getattribute__(self, "__sealed__") or name in object.__getattribute__(self, "__dict__"):
object.__getattribute__(self, "__dict__")[name] = value
else:
raise AttributeError("Attribute {} not found".format(name))
def _tear_down(self):
# Clear all remote members
for k in self._remote_attributes.keys():
self.__dict__.pop(k)
self._remote_attributes = {}
@@ -0,0 +1,108 @@
"""
Provides classes that implement the StreamSource/StreamSink and
PacketSource/PacketSink interfaces for serial ports.
"""
import os
import re
import time
import traceback
import serial
import serial.tools.list_ports
import fibre
from fibre.utils import TimeoutError
# TODO: make this customizable
DEFAULT_BAUDRATE = 115200
class SerialStreamTransport(fibre.protocol.StreamSource, fibre.protocol.StreamSink):
def __init__(self, port, baud):
self._timeout = 1
self._dev = serial.Serial(port, baud, timeout=self._timeout)
def process_bytes(self, bytes):
self._dev.write(bytes)
def get_bytes(self, n_bytes, deadline):
"""
Returns n bytes unless the deadline is reached, in which case the bytes
that were read up to that point are returned. If deadline is None the
function blocks forever. A deadline before the current time corresponds
to non-blocking mode.
"""
# Only set new timeout value if it is reasonably different from the old one (e.g. 20% as below)
# Otherwise it adds significant overhead (at least under Win10) as the port is reset with every reconfiguration
if deadline is None and self._timeout is not None:
self._timeout = None
self._dev.timeout = None
elif deadline is not None:
new_timeout = max(deadline - time.monotonic(), 0)
if abs(new_timeout - self._timeout) > self._timeout * 0.2:
self._timeout = new_timeout
self._dev.timeout = new_timeout
return self._dev.read(n_bytes)
def get_bytes_or_fail(self, n_bytes, deadline):
result = self.get_bytes(n_bytes, deadline)
if len(result) < n_bytes:
raise TimeoutError("expected {} bytes but got only {}", n_bytes, len(result))
return result
def close(self):
self._dev.close()
def find_dev_serial_ports():
try:
return ['/dev/' + x for x in os.listdir('/dev')]
except FileNotFoundError:
return []
def find_pyserial_ports():
return [x.device for x in serial.tools.list_ports.comports()]
def discover_channels(path, serial_number, callback, cancellation_token, channel_termination_token, logger):
"""
Scans for serial ports that match the path spec.
This function blocks until cancellation_token is set.
Channels spawned by this function run until channel_termination_token is set.
"""
if path == None:
# This regex should match all desired port names on macOS,
# Linux and Windows but might match some incorrect port names.
regex = r'^(/dev/tty\.usbmodem.*|/dev/ttyACM.*|COM[0-9]+)$'
else:
regex = "^" + path + "$"
known_devices = []
def device_matcher(port_name):
if port_name in known_devices:
return False
return bool(re.match(regex, port_name))
def did_disconnect(port_name, device):
device.close()
# TODO: yes there is a race condition here in case you wonder.
known_devices.pop(known_devices.index(port_name))
while not cancellation_token.is_set():
all_ports = find_pyserial_ports() + find_dev_serial_ports()
new_ports = filter(device_matcher, all_ports)
for port_name in new_ports:
try:
serial_device = SerialStreamTransport(port_name, DEFAULT_BAUDRATE)
input_stream = fibre.protocol.PacketFromStreamConverter(serial_device)
output_stream = fibre.protocol.StreamBasedPacketSink(serial_device)
channel = fibre.protocol.Channel(
"serial port {}@{}".format(port_name, DEFAULT_BAUDRATE),
input_stream, output_stream, channel_termination_token, logger)
channel.serial_device = serial_device
except serial.serialutil.SerialException:
logger.debug("Serial device init failed. Ignoring this port. More info: " + traceback.format_exc())
known_devices.append(port_name)
else:
known_devices.append(port_name)
channel._channel_broken.subscribe(lambda: did_disconnect(port_name, serial_device))
callback(channel)
time.sleep(1)
@@ -0,0 +1,128 @@
import sys
import platform
import threading
import fibre
def did_discover_device(device,
interactive_variables, discovered_devices,
branding_short, branding_long,
logger, app_shutdown_token):
"""
Handles the discovery of new devices by displaying a
message and making the device available to the interactive
console
"""
serial_number = '{:012X}'.format(device.serial_number) if hasattr(device, 'serial_number') else "[unknown serial number]"
if serial_number in discovered_devices:
verb = "Reconnected"
index = discovered_devices.index(serial_number)
else:
verb = "Connected"
discovered_devices.append(serial_number)
index = len(discovered_devices) - 1
interactive_name = branding_short + str(index)
# Publish new device to interactive console
interactive_variables[interactive_name] = device
globals()[interactive_name] = device # Add to globals so tab complete works
logger.notify("{} to {} {} as {}".format(verb, branding_long, serial_number, interactive_name))
# Subscribe to disappearance of the device
device.__channel__._channel_broken.subscribe(lambda: did_lose_device(interactive_name, logger, app_shutdown_token))
def did_lose_device(interactive_name, logger, app_shutdown_token):
"""
Handles the disappearance of a device by displaying
a message.
"""
if not app_shutdown_token.is_set():
logger.warn("Oh no {} disappeared".format(interactive_name))
def launch_shell(args,
interactive_variables,
print_banner, print_help,
logger, app_shutdown_token,
branding_short="dev", branding_long="device"):
"""
Launches an interactive python or IPython command line
interface.
As devices are connected they are made available as
"dev0", "dev1", ...
The names of the variables can be customized by setting branding_short.
"""
discovered_devices = []
globals().update(interactive_variables)
# Connect to device
logger.debug("Waiting for {}...".format(branding_long))
fibre.find_all(args.path, args.serial_number,
lambda dev: did_discover_device(dev, interactive_variables, discovered_devices, branding_short, branding_long, logger, app_shutdown_token),
app_shutdown_token,
app_shutdown_token,
logger=logger)
# Check if IPython is installed
if args.no_ipython:
use_ipython = False
else:
try:
import IPython
use_ipython = True
except:
print("Warning: you don't have IPython installed.")
print("If you want to have an improved interactive console with pretty colors,")
print("you should install IPython\n")
use_ipython = False
interactive_variables["help"] = lambda: print_help(args, len(discovered_devices) > 0)
# If IPython is installed, embed IPython shell, otherwise embed regular shell
if use_ipython:
# Override help function # pylint: disable=W0612
help = lambda: print_help(args, len(discovered_devices) > 0)
# to fix broken "%run -i script.py"
locals()['__name__'] = globals()['__name__']
console = IPython.terminal.embed.InteractiveShellEmbed(banner1='')
# hack to make IPython look like the regular console
console.runcode = console.run_cell
interact = console
# Catch ChannelBrokenException (since disconnect is not always an error)
default_exception_hook = console._showtraceback
def filtered_exception_hook(ex_class, ex, trace):
if(ex_class.__module__+'.'+ex_class.__name__ != 'fibre.protocol.ChannelBrokenException'):
default_exception_hook(ex_class,ex,trace)
console._showtraceback = filtered_exception_hook
else:
# Enable tab complete if possible
try:
import readline # Works only on Unix
readline.parse_and_bind("tab: complete")
except:
sudo_prefix = "" if platform.system() == "Windows" else "sudo "
print("Warning: could not enable tab-complete. User experience will suffer.\n"
"Run `{}pip install readline` and then restart this script to fix this."
.format(sudo_prefix))
import code
console = code.InteractiveConsole(locals=interactive_variables)
interact = lambda: console.interact(banner='')
# Catch ChannelBrokenException (since disconnect is not alway an error)
console.runcode("import sys")
console.runcode("default_exception_hook = sys.excepthook")
console.runcode("def filtered_exception_hook(ex_class, ex, trace):\n"
" if ex_class.__module__ + '.' + ex_class.__name__ != 'fibre.protocol.ChannelBrokenException':\n"
" default_exception_hook(ex_class,ex,trace)")
console.runcode("sys.excepthook=filtered_exception_hook")
# Launch shell
print_banner()
logger._skip_bottom_line = True
interact()
app_shutdown_token.set()
@@ -0,0 +1,85 @@
import sys
import socket
import time
import traceback
import fibre.protocol
from fibre.utils import wait_any, TimeoutError
def noprint(x):
pass
class TCPTransport(fibre.protocol.StreamSource, fibre.protocol.StreamSink):
def __init__(self, dest_addr, dest_port, logger):
# TODO: FIXME: use IPv6
# Problem: getaddrinfo fails if the resolver returns an
# IPv4 address, but we are using AF_INET6
#family = socket.AF_INET6 if socket.has_ipv6 else socket.AF_INET
family = socket.AF_INET
self.sock = socket.socket(family, socket.SOCK_STREAM)
# TODO: Determine the right address to use from the list
self.target = socket.getaddrinfo(dest_addr, dest_port, family)[0][4]
# TODO: this blocks until a connection is established, or the system cancels it
self.sock.connect(self.target)
def process_bytes(self, buffer):
self.sock.send(buffer)
def get_bytes(self, n_bytes, deadline):
"""
Returns n bytes unless the deadline is reached, in which case the bytes
that were read up to that point are returned. If deadline is None the
function blocks forever. A deadline before the current time corresponds
to non-blocking mode.
"""
# convert deadline to seconds (floating point)
deadline = None if deadline is None else max(deadline - time.monotonic(), 0)
self.sock.settimeout(deadline)
try:
data = self.sock.recv(n_bytes, socket.MSG_WAITALL) # receive n_bytes
return data
except socket.timeout:
# if we got a timeout data will still be none, so we call recv again
# this time in non blocking state and see if we can get some data
try:
return self.sock.recv(n_bytes, socket.MSG_DONTWAIT)
except socket.timeout:
raise TimeoutError
def get_bytes_or_fail(self, n_bytes, deadline):
result = self.get_bytes(n_bytes, deadline)
if len(result) < n_bytes:
raise TimeoutError("expected {} bytes but got only {}".format(n_bytes, len(result)))
return result
def discover_channels(path, serial_number, callback, cancellation_token, channel_termination_token, logger):
"""
Tries to connect to a TCP server based on the path spec.
This function blocks until cancellation_token is set.
Channels spawned by this function run until channel_termination_token is set.
"""
try:
dest_addr = ':'.join(path.split(":")[:-1])
dest_port = int(path.split(":")[-1])
except (ValueError, IndexError):
raise Exception('"{}" is not a valid TCP destination. The format should be something like "localhost:1234".'
.format(path))
while not cancellation_token.is_set():
try:
tcp_transport = fibre.tcp_transport.TCPTransport(dest_addr, dest_port, logger)
stream2packet_input = fibre.protocol.PacketFromStreamConverter(tcp_transport)
packet2stream_output = fibre.protocol.StreamBasedPacketSink(tcp_transport)
channel = fibre.protocol.Channel(
"TCP device {}:{}".format(dest_addr, dest_port),
stream2packet_input, packet2stream_output,
channel_termination_token, logger)
except:
#logger.debug("TCP channel init failed. More info: " + traceback.format_exc())
pass
else:
callback(channel)
wait_any(None, cancellation_token, channel._channel_broken)
time.sleep(1)
@@ -0,0 +1,57 @@
import sys
import socket
import time
import traceback
import fibre.protocol
from fibre.utils import wait_any
def noprint(x):
pass
class UDPTransport(fibre.protocol.PacketSource, fibre.protocol.PacketSink):
def __init__(self, dest_addr, dest_port, logger):
# TODO: FIXME: use IPv6
# Problem: getaddrinfo fails if the resolver returns an
# IPv4 address, but we are using AF_INET6
#family = socket.AF_INET6 if socket.has_ipv6 else socket.AF_INET
family = socket.AF_INET
self.sock = socket.socket(family, socket.SOCK_DGRAM)
# TODO: Determine the right address to use from the list
self.target = socket.getaddrinfo(dest_addr,dest_port, family)[0][4]
def process_packet(self, buffer):
self.sock.sendto(buffer, self.target)
def get_packet(self, deadline):
# TODO: implement deadline
data, _ = self.sock.recvfrom(1024)
return data
def discover_channels(path, serial_number, callback, cancellation_token, channel_termination_token, logger):
"""
Tries to connect to a UDP server based on the path spec.
This function blocks until cancellation_token is set.
Channels spawned by this function run until channel_termination_token is set.
"""
try:
dest_addr = ':'.join(path.split(":")[:-1])
dest_port = int(path.split(":")[-1])
except (ValueError, IndexError):
raise Exception('"{}" is not a valid UDP destination. The format should be something like "localhost:1234".'
.format(path))
while not cancellation_token.is_set():
try:
udp_transport = fibre.udp_transport.UDPTransport(dest_addr, dest_port, logger)
channel = fibre.protocol.Channel(
"UDP device {}:{}".format(dest_addr, dest_port),
udp_transport, udp_transport,
channel_termination_token, logger)
except:
logger.debug("UDP channel init failed. More info: " + traceback.format_exc())
pass
else:
callback(channel)
wait_any(None, cancellation_token, channel._channel_broken)
time.sleep(1)
@@ -0,0 +1,216 @@
# requires pyusb
# pip install --pre pyusb
import usb.core
import usb.util
import sys
import time
import fibre.protocol
import traceback
import platform
from fibre.utils import TimeoutError
# Currently we identify fibre-enabled devices by VID,PID
# TODO: identify by USB descriptors
WELL_KNOWN_VID_PID_PAIRS = [
(0x1209, 0x0D31),
(0x1209, 0x0D32),
(0x1209, 0x0D33)
]
class USBBulkTransport(fibre.protocol.PacketSource, fibre.protocol.PacketSink):
def __init__(self, dev, logger):
self._logger = logger
self.dev = dev
self.intf = None
self._name = "USB device {}:{}".format(dev.idVendor, dev.idProduct)
self._was_damaged = False
##
# information about the connected device
##
def info(self):
# loop through configurations
string = ""
for cfg in self.dev:
string += "ConfigurationValue {0}\n".format(cfg.bConfigurationValue)
for intf in cfg:
string += "\tInterfaceNumber {0},{1}\n".format(intf.bInterfaceNumber, intf.bAlternateSetting)
for ep in intf:
string += "\t\tEndpointAddress {0}\n".format(ep.bEndpointAddress)
return string
def init(self):
# Under some conditions, the Linux USB/libusb stack ends up in a corrupt
# state where there are a few packets in a receive queue but a call
# to epr.read() does not return these packet until a new packet arrives.
# This undesirable queue can be cleared by resetting the device.
# On windows this would cause file-not-found errors in subsequent dev calls
if platform.system() != 'Windows':
self.dev.reset()
#self.dev.set_configuration() # no args: set first configuration
# Find the best interface
self.cfg = self.dev.get_active_configuration()
custom_interfaces = [i for i in self.cfg.interfaces() if i.bInterfaceClass == 0x00 and i.bInterfaceSubClass == 0x01]
cdc_interfaces = [i for i in self.cfg.interfaces() if i.bInterfaceClass == 0x0a and i.bInterfaceSubClass == 0x00]
all_compatible_interfaces = custom_interfaces + cdc_interfaces
if len(all_compatible_interfaces) == 0:
raise Exception("the device has no compatible interfaces")
self.intf = all_compatible_interfaces[0]
# Try to detach kernel driver from interface
try:
if self.dev.is_kernel_driver_active(self.intf.bInterfaceNumber):
self.dev.detach_kernel_driver(self.intf.bInterfaceNumber)
self._logger.debug("Detached Kernel Driver")
else:
self._logger.debug("Kernel Driver was not attached")
except NotImplementedError:
pass #is_kernel_driver_active not implemented on Windows
# find write endpoint (first OUT endpoint)
self.epw = usb.util.find_descriptor(self.intf,
custom_match = \
lambda e: \
usb.util.endpoint_direction(e.bEndpointAddress) == \
usb.util.ENDPOINT_OUT
)
assert self.epw is not None
self._logger.debug("EndpointAddress for writing {}".format(self.epw.bEndpointAddress))
# find read endpoint (first IN endpoint)
self.epr = usb.util.find_descriptor(self.intf,
custom_match = \
lambda e: \
usb.util.endpoint_direction(e.bEndpointAddress) == \
usb.util.ENDPOINT_IN
)
assert self.epr is not None
self._logger.debug("EndpointAddress for reading {}".format(self.epr.bEndpointAddress))
def deinit(self):
if not self.intf is None:
usb.util.release_interface(self.dev, self.intf)
def process_packet(self, usbBuffer):
try:
ret = self.epw.write(usbBuffer, 0)
if self._was_damaged:
self._logger.debug("Recovered from USB halt/stall condition")
self._was_damaged = False
return ret
except usb.core.USBError as ex:
if ex.errno == 19 or ex.errno == 32: # "no such device", "pipe error"
raise fibre.protocol.ChannelBrokenException()
elif ex.errno is None or ex.errno == 60 or ex.errno == 110: # timeout
raise TimeoutError()
else:
self._logger.debug("error in usbbulk_transport.py, process_packet")
self._logger.debug(traceback.format_exc())
self._logger.debug("halt condition: {}".format(ex.errno))
self._logger.debug(str(ex))
# Try resetting halt/stall condition
try:
self.deinit()
self.init()
except usb.core.USBError:
raise fibre.protocol.ChannelBrokenException()
# Retry transfer
self._was_damaged = True
raise fibre.protocol.ChannelDamagedException()
def get_packet(self, deadline):
try:
bufferLen = self.epr.wMaxPacketSize
timeout = max(int((deadline - time.monotonic()) * 1000), 0)
ret = self.epr.read(bufferLen, timeout)
if self._was_damaged:
self._logger.debug("Recovered from USB halt/stall condition")
self._was_damaged = False
return bytearray(ret)
except usb.core.USBError as ex:
if ex.errno == 19 or ex.errno == 32: # "no such device", "pipe error"
raise fibre.protocol.ChannelBrokenException()
elif ex.errno is None or ex.errno == 60 or ex.errno == 110: # timeout
raise TimeoutError()
else:
self._logger.debug("error in usbbulk_transport.py, process_packet")
self._logger.debug(traceback.format_exc())
self._logger.debug("halt condition: {}".format(ex.errno))
self._logger.debug(str(ex))
# Try resetting halt/stall condition
try:
self.deinit()
self.init()
except usb.core.USBError:
raise fibre.protocol.ChannelBrokenException()
# Retry transfer
self._was_damaged = True
raise fibre.protocol.ChannelDamagedException()
def discover_channels(path, serial_number, callback, cancellation_token, channel_termination_token, logger):
"""
Scans for USB devices that match the path spec.
This function blocks until cancellation_token is set.
Channels spawned by this function run until channel_termination_token is set.
"""
if path == None or path == "":
bus = None
address = None
else:
try:
bus = int(path.split(":")[0])
address = int(path.split(":")[1])
except (ValueError, IndexError):
raise Exception("{} is not a valid USB path specification. "
"Expected a string of the format BUS:DEVICE where BUS "
"and DEVICE are integers.".format(path))
known_devices = []
def device_matcher(device):
#print(" test {:04X}:{:04X}".format(device.idVendor, device.idProduct))
try:
if (device.bus, device.address) in known_devices:
return False
if bus != None and device.bus != bus:
return False
if address != None and device.address != address:
return False
if serial_number != None and device.serial_number != serial_number:
return False
if (device.idVendor, device.idProduct) not in WELL_KNOWN_VID_PID_PAIRS:
return False
except:
return False
return True
while not cancellation_token.is_set():
# logger.debug("USB discover loop")
devices = usb.core.find(find_all=True, custom_match=device_matcher)
for usb_device in devices:
try:
bulk_device = USBBulkTransport(usb_device, logger)
logger.debug(bulk_device.info())
bulk_device.init()
channel = fibre.protocol.Channel(
"USB device bus {} device {}".format(usb_device.bus, usb_device.address),
bulk_device, bulk_device, channel_termination_token, logger)
channel.usb_device = usb_device # for debugging only
except usb.core.USBError as ex:
if ex.errno == 13:
# TODO: this is an ODrive specific message and should live outside of the fibre library
logger.warn("I found a USB device that looks like an ODrive (bus {}, device {}) but I can't access it. Try running `sudo odrivetool udev-setup`, then unplug and replug the device.".format(usb_device.bus, usb_device.address))
known_devices.append((usb_device.bus, usb_device.address))
elif ex.errno == 16:
logger.debug("USB device busy. I'll reset it and try again.")
usb_device.reset()
continue
else:
logger.warn("USB device init failed (bus {}, device {}). Ignoring this device. More info: ".format(usb_device.bus, usb_device.address) + traceback.format_exc())
known_devices.append((usb_device.bus, usb_device.address))
else:
known_devices.append((usb_device.bus, usb_device.address))
callback(channel)
time.sleep(1)
@@ -0,0 +1,237 @@
import sys
import time
import threading
import platform
import subprocess
import os
try:
if platform.system() == 'Windows':
import win32console
# TODO: we should win32console anyway so we could just omit colorama
import colorama
colorama.init()
except ImportError:
print("Could not init terminal features.")
sys.stdout.flush()
pass
if sys.version_info < (3, 3):
class TimeoutError(Exception):
pass
else:
TimeoutError = TimeoutError
def get_serial_number_str(device):
if hasattr(device, 'serial_number'):
return format(device.serial_number, 'x').upper()
else:
return "[unknown serial number]"
## Threading utils ##
class Event():
"""
Alternative to threading.Event(), enhanced by the subscribe() function
that the original fails to provide.
@param Trigger: if supplied, the newly created event will be triggered
as soon as the trigger event becomes set
"""
def __init__(self, trigger=None):
self._evt = threading.Event()
self._subscribers = []
self._mutex = threading.Lock()
if not trigger is None:
trigger.subscribe(lambda: self.set())
def is_set(self):
return self._evt.is_set()
def set(self):
"""
Sets the event and invokes all subscribers if the event was
not already set
"""
self._mutex.acquire()
try:
if not self._evt.is_set():
self._evt.set()
for s in self._subscribers:
s()
finally:
self._mutex.release()
def subscribe(self, handler):
"""
Invokes the specified handler exactly once as soon as the
specified event is set. If the event is already set, the
handler is invoked immediately.
Returns a function that can be invoked to unsubscribe.
"""
if handler is None:
raise TypeError
self._mutex.acquire()
try:
self._subscribers.append(handler)
if self._evt.is_set():
handler()
finally:
self._mutex.release()
return handler
def unsubscribe(self, handler):
self._mutex.acquire()
try:
self._subscribers.pop(self._subscribers.index(handler))
finally:
self._mutex.release()
def wait(self, timeout=None):
if not self._evt.wait(timeout=timeout):
raise TimeoutError()
def trigger_after(self, timeout):
"""
Triggers the event after the specified timeout.
This function returns immediately.
"""
def delayed_trigger():
if not self.wait(timeout=timeout):
self.set()
threading.Thread(target=delayed_trigger)
t.daemon = True
t.start()
def wait_any(timeout=None, *events):
"""
Blocks until any of the specified events are triggered.
Returns the index of the event that was triggerd or raises
a TimeoutError
Param timeout: A timeout in seconds
"""
or_event = threading.Event()
subscriptions = []
for event in events:
subscriptions.append((event, event.subscribe(lambda: or_event.set())))
or_event.wait(timeout=timeout)
for event, sub in subscriptions:
event.unsubscribe(sub)
for i in range(len(events)):
if events[i].is_set():
return i
raise TimeoutError()
## Log utils ##
class Logger():
"""
Logs messages to stdout
"""
COLOR_DEFAULT = 0
COLOR_GREEN = 1
COLOR_CYAN = 2
COLOR_YELLOW = 3
COLOR_RED = 4
_VT100Colors = {
COLOR_GREEN: '\x1b[92;1m',
COLOR_CYAN: '\x1b[96;1m',
COLOR_YELLOW: '\x1b[93;1m',
COLOR_RED: '\x1b[91;1m',
COLOR_DEFAULT: '\x1b[0m'
}
_Win32Colors = {
COLOR_GREEN: 0x0A,
COLOR_CYAN: 0x0B,
COLOR_YELLOW: 0x0E,
COLOR_RED: 0x0C,
COLOR_DEFAULT: 0x07
}
def __init__(self, verbose=True):
self._prefix = ''
self._skip_bottom_line = False # If true, messages are printed one line above the cursor
self._verbose = verbose
self._print_lock = threading.Lock()
if platform.system() == 'Windows':
self._stdout_buf = win32console.GetStdHandle(win32console.STD_OUTPUT_HANDLE)
def indent(self, prefix=' '):
indented_logger = Logger()
indented_logger._prefix = self._prefix + prefix
return indented_logger
def print_on_second_last_line(self, text, color):
"""
Prints a text on the second last line.
This can be used to print a message above the command
prompt. If the command prompt spans multiple lines
there will be glitches.
If the printed text spans multiple lines there will also
be glitches (though this could be fixed).
"""
if platform.system() == 'Windows':
# Windows <10 doesn't understand VT100 escape codes and the colorama
# also doesn't support the specific escape codes we need so we use the
# native Win32 API.
info = self._stdout_buf.GetConsoleScreenBufferInfo()
cursor_pos = info['CursorPosition']
scroll_rect=win32console.PySMALL_RECTType(
Left=0, Top=1,
Right=info['Window'].Right,
Bottom=cursor_pos.Y-1)
scroll_dest = win32console.PyCOORDType(scroll_rect.Left, scroll_rect.Top-1)
self._stdout_buf.ScrollConsoleScreenBuffer(
scroll_rect, scroll_rect, scroll_dest, # clipping rect is same as scroll rect
u' ', Logger._Win32Colors[color]) # fill with empty cells with the desired color attributes
line_start = win32console.PyCOORDType(0, cursor_pos.Y-1)
self._stdout_buf.WriteConsoleOutputCharacter(text, line_start)
else:
# Assume we're in a terminal that interprets VT100 escape codes.
# TODO: test on macOS
# Escape character sequence:
# ESC 7: store cursor position
# ESC 1A: move cursor up by one
# ESC 1S: scroll entire viewport by one
# ESC 1L: insert 1 line at cursor position
# (print text)
# ESC 8: restore old cursor position
self._print_lock.acquire()
sys.stdout.write('\x1b7\x1b[1A\x1b[1S\x1b[1L')
sys.stdout.write(Logger._VT100Colors[color] + text + Logger._VT100Colors[Logger.COLOR_DEFAULT])
sys.stdout.write('\x1b8')
sys.stdout.flush()
self._print_lock.release()
def print_colored(self, text, color):
if self._skip_bottom_line:
self.print_on_second_last_line(text, color)
else:
# On Windows, colorama does the job of interpreting the VT100 escape sequences
self._print_lock.acquire()
sys.stdout.write(Logger._VT100Colors[color] + text + Logger._VT100Colors[Logger.COLOR_DEFAULT] + '\n')
sys.stdout.flush()
self._print_lock.release()
def debug(self, text):
if self._verbose:
self.print_colored(self._prefix + text, Logger.COLOR_DEFAULT)
def success(self, text):
self.print_colored(self._prefix + text, Logger.COLOR_GREEN)
def info(self, text):
self.print_colored(self._prefix + text, Logger.COLOR_DEFAULT)
def notify(self, text):
self.print_colored(self._prefix + text, Logger.COLOR_CYAN)
def warn(self, text):
self.print_colored(self._prefix + text, Logger.COLOR_YELLOW)
def error(self, text):
# TODO: write to stderr
self.print_colored(self._prefix + text, Logger.COLOR_RED)
@@ -0,0 +1,90 @@
"""
This script is used to deploy the Fibre python library to PyPi
so that users can install them easily with
"pip install fibre"
To install the package and its dependencies locally, run:
sudo pip install -r requirements.txt
To build and package the python tools into a tar archive:
python setup.py sdist
Warning: Before you proceed, be aware that you can upload a
specific version only once ever. After that you need to increment
the hotfix number. Deleting the release manually on the PyPi
website does not help.
Use TestPyPi while developing.
To build, package and upload the python tools to TestPyPi, run:
python setup.py sdist upload -r pypitest
To make a real release ensure you're at the release commit
and then run the above command without the "test" (so just "pypi").
To install a prerelease version from test index:
sudo pip install --pre --index-url https://test.pypi.org/simple/ --no-cache-dir fibre
PyPi access requires that you have set up ~/.pypirc with your
PyPi credentials and that your account has the rights
to publish packages with the name fibre.
"""
# TODO: add additional y/n prompt to prevent from erroneous upload
from setuptools import setup
import os
import sys
# Change this if you already uploaded the current
# version but need to release a hotfix
hotfix = 0
#creating_package = "sdist" in sys.argv
#
## Load version from Git tag
#import odrive.version
#version = odrive.version.get_version_str(git_only=creating_package)
#
#if creating_package and (hotfix > 0 or not version[-1].isdigit()):
# # Add this for hotfixes
# version += "-" + str(hotfix)
#
#
## If we're currently creating the package we need to autogenerate
## a file that contains the version string
#if creating_package:
# version_file_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'odrive', 'version.txt')
# with open(version_file_path, mode='w') as version_file:
# version_file.write(version)
#
## TODO: find a better place for this
#if not creating_package:
# import platform
# if platform.system() == 'Linux':
# import odrive.utils
# odrive.utils.setup_udev_rules(odrive.utils.Logger())
setup(
name = 'fibre',
packages = ['fibre'],
#scripts = ['..fibre', 'odrivetool.bat', 'odrive_demo.py'],
version = '0.0.1dev0',
description = 'Abstraction layer for painlessly building object oriented distributed systems that just work',
author = 'Samuel Sadok',
author_email = 'samuel.sadok@bluewin.ch',
license='MIT',
url = 'https://github.com/samuelsadok/fibre',
keywords = ['communication', 'transport-layer', 'rpc'],
install_requires = [
'appdirs', # Used to find caching directory
],
#package_data={'': ['version.txt']},
classifiers = [],
)
# TODO: include README
## clean up
#if creating_package:
# os.remove(version_file_path)
@@ -0,0 +1,27 @@
tup.include('../tupfiles/build.lua')
tup.include('../cpp/package.lua')
test_server = define_package{
packages={fibre_package},
sources={'test_server.cpp'}
}
unit_tests = define_package{
packages={fibre_package},
sources={'run_tests.cpp'}
}
toolchain=GCCToolchain('', 'build', {'-O3', '-fvisibility=hidden', '-frename-registers', '-funroll-loops'}, {})
toolchain=GCCToolchain('', 'build', {'-O3', '-g', '-Wall'}, {})
--toolchain=GCCToolchain('avr-', {'-Ofast', '-fvisibility=hidden', '-frename-registers', '-funroll-loops', '-I/home/samuel/stlport-avr/stlport'}, {})
--toolchain=LLVMToolchain('x86_64', {'-O3', '-fno-sanitize=safe-stack', '-fno-stack-protector'}, {'-flto', '-Wl,-s'})
--toolchain=LLVMToolchain('avr', {'-O3', '-std=gnu++11', '--target=avr', '-fno-sanitize=safe-stack', '-fno-stack-protector', '-I/home/samuel/stlport-avr/stlport'}, {'-flto', '-Wl,-s'})
if tup.getconfig("BUILD_FIBRE_TESTS") == "true" then
build_executable('test_server', test_server, toolchain)
--build_executable('run_tests', unit_tests, toolchain)
end
@@ -0,0 +1,140 @@
#include <stdint.h>
#include <stdlib.h>
#include <limits.h>
#include <stdio.h>
//#define DEBUG_PROTOCOL
void hexdump(const uint8_t* buf, size_t len);
#include <fibre/crc.hpp>
#include <fibre/decoders.hpp>
#include <fibre/encoders.hpp>
void hexdump(const uint8_t* buf, size_t len) {
for (size_t pos = 0; pos < len; ++pos) {
printf(" %02x", buf[pos]);
if ((((pos + 1) % 16) == 0) || ((pos + 1) == len))
printf("\r\n");
//osDelay(2);
}
}
bool varint_decoder_test() {
struct test_case_t {
uint8_t encoded[10];
size_t length;
uint32_t decoded;
};
const test_case_t test_cases[] = {
// encoded, length, decoded
{ { 0x00 }, 1, 0 },
{ { 0x01 }, 1, 1 },
{ { 0xff, 0x01 }, 2, 0xff },
{ { 0xAC, 0x02 }, 2, 300 },
{ { 0xff, 0xff, 0xff, 0xff, 0xf }, 5, 0xffffffff }
};
for (size_t i = 0; i < sizeof(test_cases) / sizeof(test_cases[0]); ++i) {
uint32_t result;
VarintStreamDecoder<uint32_t> decoder = make_varint_decoder(result);
size_t processed_bytes = 0;
int status = decoder.process_bytes(test_cases[i].encoded, test_cases[i].length, &processed_bytes);
if (status) {
return false;
} else if (processed_bytes != test_cases[i].length) {
printf("test %zu: expected to process %zu bytes but processed %zu bytes\n", i, test_cases[i].length, processed_bytes);
return false;
} else if (result != test_cases[i].decoded) {
printf("test %zu: expected %u but got %u\n", i, test_cases[i].decoded, result);
return false;
}
VarintStreamEncoder<uint32_t> encoder = make_varint_encoder(test_cases[i].decoded);
uint8_t buffer[10];
size_t generated_bytes = 0;
status = encoder.get_bytes(buffer, sizeof(buffer), &generated_bytes);
if (status) {
return false;
} else if ((generated_bytes != test_cases[i].length)
|| memcmp(buffer, test_cases[i].encoded, test_cases[i].length)) {
printf("test %zu: expected:", i);
hexdump(test_cases[i].encoded, test_cases[i].length);
printf("got: ");
hexdump(buffer, generated_bytes);
return false;
}
}
return true;
}
int main(void) {
/***** Decoder demo (remove or move somewhere else) *****/
printf("Running decoder... ");
// prepare raw data
uint8_t raw_data[] = { 0xBC, 0x03, 0xAC, 0x5e, 0x02, 0x00, 0x00, 0xd1 };
//raw_data[3] = calc_crc8<CANONICAL_CRC8_POLYNOMIAL>(CANONICAL_CRC8_INIT, raw_data, 3);
//raw_data[7] = calc_crc8<CANONICAL_CRC8_POLYNOMIAL>(raw_data[3], raw_data + 4, 3);
// instantiate decoder
ReceiverState state;
auto decoder = make_crc8_decoder<CANONICAL_CRC8_INIT, CANONICAL_CRC8_POLYNOMIAL>(
make_decoder_chain(
make_length_decoder(state),
make_endpoint_id_decoder(state)
)
);
// push the raw data through the decoder
size_t processed_bytes = 0;
int status = decoder.process_bytes(raw_data, sizeof(raw_data), &processed_bytes);
// expected result: "length: 444, endpoint-id: 300, processed 8 bytes"
if (status == 0)
printf("length: %zu, endpoint-id: %zu, processed %zu bytes\n", state.length, state.endpoint_id, processed_bytes);
else
printf("decoder demo failed\n");
/***** Encoder demo (remove or move somewhere else) *****/
printf("Running encoder... ");
// prepare request
Request request = {
.endpoint_id = 300,
.length = 444,
};
// construct encoder for the request
auto e2 = make_crc8_encoder<CANONICAL_CRC8_INIT, CANONICAL_CRC8_POLYNOMIAL>(
make_encoder_chain(
make_length_encoder(request),
make_endpoint_id_encoder(request)
)
);
// pull raw data out of the encoder
uint8_t buffer[20];
size_t generated_bytes = 0;
status = e2.get_bytes(buffer, sizeof(buffer), &generated_bytes);
if (status == 0) {
printf("generated %zu bytes:\n", generated_bytes);
hexdump(buffer, generated_bytes);
} else {
printf("encoder demo failed\n");
}
/***** run automated test *****/
bool test_result = varint_decoder_test();
if (test_result) {
printf("all tests passed\n");
return 0;
} else {
printf("some tests failed\n");
return -1;
}
}
@@ -0,0 +1,52 @@
#include <stdio.h>
#include <unistd.h>
#include <thread>
#include <signal.h>
#include <fibre/protocol.hpp>
#include <fibre/posix_tcp.hpp>
#include <fibre/posix_udp.hpp>
class TestClass {
public:
float property1;
float property2;
float set_both(float arg1, float arg2) {
property1 = arg1;
property2 = arg2;
return property1 + property2;
}
FIBRE_EXPORTS(TestClass,
make_protocol_property("property1", &property1),
make_protocol_property("property2", &property2),
make_protocol_function("set_both", *obj, &TestClass::set_both, "arg1", "arg2")
);
};
int main() {
printf("Starting Fibre server...\n");
TestClass test_object = TestClass();
// publish the object on Fibre
auto definitions = test_object.fibre_definitions;
fibre_publish(definitions);
// Expose Fibre objects on TCP and UDP
std::thread server_thread_tcp(serve_on_tcp, 9910);
std::thread server_thread_udp(serve_on_udp, 9910);
printf("Fibre server started.\n");
// Dump property1 value
while (1) {
printf("test_object.property1: %f\n", test_object.property1);
usleep(1000000 / 5); // 5 Hz
}
return 0;
}
@@ -0,0 +1,57 @@
#!/usr/bin/env python3
"""
Connect to a Fibre-enabled device to play with in the IPython interactive shell.
"""
import argparse
import sys
import os
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.realpath(__file__))) + "/python")
from fibre import Logger, Event
# Parse arguments
parser = argparse.ArgumentParser(description='Connect to a fibre-enabled device to play with it in the IPython interactive shell.')
parser.add_argument("-p", "--path", metavar="PATH", action="store",
help="The path(s) where ODrive(s) should be discovered.\n"
"By default the script will connect to any ODrive on USB.\n\n"
"To select a specific USB device:\n"
" --path usb:BUS:DEVICE\n"
"usbwhere BUS and DEVICE are the bus and device numbers as shown in `lsusb`.\n\n"
"To select a specific serial port:\n"
" --path serial:PATH\n"
"where PATH is the path of the serial port. For example \"/dev/ttyUSB0\".\n"
"You can use `ls /dev/tty*` to find the correct port.\n\n"
"You can combine USB and serial specs by separating them with a comma (no space!)\n"
"Example:\n"
" --path usb,serial:/dev/ttyUSB0\n"
"means \"discover any USB device or a serial device on /dev/ttyUSB0\"")
parser.add_argument("-s", "--serial-number", action="store",
help="The 12-digit serial number of the device. "
"This is a string consisting of 12 upper case hexadecimal "
"digits as displayed in lsusb. \n"
" example: 385F324D3037\n"
"You can list all devices connected to USB by running\n"
"(lsusb -d 1209:0d32 -v; lsusb -d 0483:df11 -v) | grep iSerial\n"
"If omitted, any device is accepted.")
parser.add_argument("--no-ipython", action="store_true",
help="Use the regular Python shell "
"instead of the IPython shell, "
"even if IPython is installed.")
parser.add_argument("-v", "--verbose", action="store_true",
help="print debug information")
parser.set_defaults(path="usb,tcp:localhost:9910")
args = parser.parse_args()
logger = Logger(verbose=args.verbose)
app_shutdown_token = Event()
def print_banner():
pass
def print_help(args, have_devices):
pass
import fibre
fibre.launch_shell(args, {}, print_banner, print_help, logger, app_shutdown_token)
@@ -0,0 +1,685 @@
#!/bin/python3
import yaml
import json
import jinja2
import jsonschema
import re
import argparse
import sys
from collections import OrderedDict
# This schema describes what we expect interface definition files to look like
validator = jsonschema.Draft4Validator(yaml.safe_load("""
definitions:
interface:
type: object
properties:
c_is_class: {type: boolean}
c_name: {type: string}
brief: {type: string}
doc: {type: string}
functions:
type: object
additionalProperties: {"$ref": "#/definitions/function"}
attributes:
type: object
additionalProperties: {"$ref": "#/definitions/attribute"}
__line__: {type: object}
__column__: {type: object}
required: [c_is_class]
additionalProperties: false
valuetype:
type: object
properties:
mode: {type: string} # this shouldn't be here
c_name: {type: string}
values: {type: object}
flags: {type: object}
nullflag: {type: string}
__line__: {type: object}
__column__: {type: object}
additionalProperties: false
intf_or_val_type:
anyOf:
- {"$ref": "#/definitions/interface"}
- {"$ref": "#/definitions/valuetype"}
- {"type": "string"}
attribute:
anyOf: # this is probably not being used correctly
- {"$ref": "#/definitions/intf_or_val_type"}
- type: object
- type: object
properties:
type: {"$ref": "#/definitions/intf_or_val_type"}
c_name: {"type": string}
unit: {"type": string}
doc: {"type": string}
additionalProperties: false
function:
anyOf:
- type: 'null'
- type: object
properties:
in: {type: object}
out: {type: object}
brief: {type: string}
doc: {type: string}
__line__: {type: object}
__column__: {type: object}
additionalProperties: false
type: object
properties:
ns: {type: string}
version: {type: string}
summary: {type: string}
dictionary: {type: array, items: {type: string}}
interfaces:
type: object
additionalProperties: { "$ref": "#/definitions/interface" }
valuetypes:
type: object
additionalProperties: { "$ref": "#/definitions/valuetype" }
__line__: {type: object}
__column__: {type: object}
additionalProperties: false
"""))
# Source: https://stackoverflow.com/a/53647080/3621512
class SafeLineLoader(yaml.SafeLoader):
pass
# def compose_node(self, parent, index):
# # the line number where the previous token has ended (plus empty lines)
# line = self.line
# node = super(SafeLineLoader, self).compose_node(parent, index)
# node.__line__ = line + 1
# return node
#
# def construct_mapping(self, node, deep=False):
# mapping = super(SafeLineLoader, self).construct_mapping(node, deep=deep)
# mapping['__line__'] = node.__line__
# #mapping['__column__'] = node.start_mark.column + 1
# return mapping
# Ensure that dicts remain ordered, even in Python <3.6
# source: https://stackoverflow.com/a/21912744/3621512
def construct_mapping(loader, node):
loader.flatten_mapping(node)
return OrderedDict(loader.construct_pairs(node))
SafeLineLoader.add_constructor(yaml.resolver.BaseResolver.DEFAULT_MAPPING_TAG, construct_mapping)
dictionary = []
def get_words(string):
"""
Splits a string in PascalCase into a list of lower case words
"""
regex = ''.join((re.escape(w) + '|') for w in dictionary) + '[a-z0-9]+|[A-Z][a-z0-9]*'
return [(w if w in dictionary else w.lower()) for w in re.findall(regex, string)]
def join_name(*names, delimiter: str = '.'):
"""
Joins two name components.
e.g. 'io.helloworld' + 'sayhello' => 'io.helloworld.sayhello'
"""
return delimiter.join(y for x in names for y in x.split(delimiter) if y != '')
def split_name(name, delimiter: str = '.'):
def replace_delimiter_in_parentheses():
parenthesis_depth = 0
for c in name:
parenthesis_depth += 1 if c == '<' else -1 if c == '>' else 0
yield c if (parenthesis_depth == 0) or (c != delimiter) else ':'
return [part.replace(':', '.') for part in ''.join(replace_delimiter_in_parentheses()).split('.')]
def to_pascal_case(s): return ''.join([(w.title() if not w in dictionary else w) for w in get_words(s)])
def to_camel_case(s): return ''.join([(c.lower() if i == 0 else c) for i, c in enumerate(''.join([w.title() for w in get_words(s)]))])
def to_macro_case(s): return '_'.join(get_words(s)).upper()
def to_snake_case(s): return '_'.join(get_words(s)).lower()
def to_kebab_case(s): return '-'.join(get_words(s)).lower()
value_types = OrderedDict({
'bool': {'builtin': True, 'fullname': 'bool', 'name': 'bool', 'c_name': 'bool', 'py_type': 'bool'},
'float32': {'builtin': True, 'fullname': 'float32', 'name': 'float32', 'c_name': 'float', 'py_type': 'float'},
'uint8': {'builtin': True, 'fullname': 'uint8', 'name': 'uint8', 'c_name': 'uint8_t', 'py_type': 'int'},
'uint16': {'builtin': True, 'fullname': 'uint16', 'name': 'uint16', 'c_name': 'uint16_t', 'py_type': 'int'},
'uint32': {'builtin': True, 'fullname': 'uint32', 'name': 'uint32', 'c_name': 'uint32_t', 'py_type': 'int'},
'uint64': {'builtin': True, 'fullname': 'uint64', 'name': 'uint64', 'c_name': 'uint64_t', 'py_type': 'int'},
'int8': {'builtin': True, 'fullname': 'int8', 'name': 'int8', 'c_name': 'int8_t', 'py_type': 'int'},
'int16': {'builtin': True, 'fullname': 'int16', 'name': 'int16', 'c_name': 'int16_t', 'py_type': 'int'},
'int32': {'builtin': True, 'fullname': 'int32', 'name': 'int32', 'c_name': 'int32_t', 'py_type': 'int'},
'int64': {'builtin': True, 'fullname': 'int64', 'name': 'int64', 'c_name': 'int64_t', 'py_type': 'int'},
'endpoint_ref': {'builtin': True, 'fullname': 'endpoint_ref', 'name': 'endpoint_ref', 'c_name': 'endpoint_ref_t', 'py_type': '[not implemented]'},
})
enums = OrderedDict()
interfaces = OrderedDict()
def make_property_type(typeargs):
value_type = resolve_valuetype('', typeargs['fibre.Property.type'])
mode = typeargs.get('fibre.Property.mode', 'readwrite')
name = 'Property<' + value_type['fullname'] + ', ' + mode + '>'
fullname = join_name('fibre', name)
if fullname in interfaces:
return interfaces[fullname]
c_name = 'Property<' + ('const ' if mode == 'readonly' else '') + value_type['c_name'] + '>'
prop_type = {
'name': name,
'fullname': fullname,
'purename': 'fibre.Property',
'c_name': c_name,
'value_type': value_type, # TODO: should be a metaarg
'mode': mode, # TODO: should be a metaarg
'builtin': True,
'attributes': OrderedDict(),
'functions': OrderedDict()
}
if mode != 'readonly':
prop_type['functions']['exchange'] = {
'name': 'exchange',
'fullname': join_name(fullname, 'exchange'),
'in': OrderedDict([('obj', {'name': 'obj', 'type': {'c_name': c_name}}), ('value', {'name': 'value', 'type': value_type, 'optional': True})]),
'out': OrderedDict([('value', {'name': 'value', 'type': value_type})]),
#'implementation': 'fibre_property_exchange<' + value_type['c_name'] + '>'
}
else:
prop_type['functions']['read'] = {
'name': 'read',
'fullname': join_name(fullname, 'read'),
'in': OrderedDict([('obj', {'name': 'obj', 'type': {'c_name': c_name}})]),
'out': OrderedDict([('value', {'name': 'value', 'type': value_type})]),
#'implementation': 'fibre_property_read<' + value_type['c_name'] + '>'
}
interfaces[fullname] = prop_type
return prop_type
generics = {
'fibre.Property': make_property_type # TODO: improve generic support
}
def make_ref_type(interface):
name = 'Ref<' + interface['fullname'] + '>'
fullname = join_name('fibre', name)
if fullname in interfaces:
return interfaces[fullname]
ref_type = {
'builtin': True,
'name': name,
'fullname': fullname,
'c_name': interface['fullname'].replace('.', 'Intf::') + 'Intf*'
}
value_types[fullname] = ref_type
return ref_type
def get_dict(elem, key):
return elem.get(key, None) or OrderedDict()
def regularize_arg(path, name, elem):
if elem is None:
elem = {}
elif isinstance(elem, str):
elem = {'type': elem}
elem['name'] = name
elem['fullname'] = path = join_name(path, name)
elem['type'] = regularize_valuetype(path, name, elem['type'])
return elem
def regularize_func(path, name, elem, prepend_args):
if elem is None:
elem = {}
elem['name'] = name
elem['fullname'] = path = join_name(path, name)
elem['in'] = OrderedDict((n, regularize_arg(path, n, arg))
for n, arg in (*prepend_args.items(), *get_dict(elem, 'in').items()))
elem['out'] = OrderedDict((n, regularize_arg(path, n, arg))
for n, arg in get_dict(elem, 'out').items())
return elem
def regularize_attribute(parent, name, elem, c_is_class):
if elem is None:
elem = {}
if isinstance(elem, str):
elem = {'type': elem}
elif not 'type' in elem:
elem['type'] = {}
if 'attributes' in elem: elem['type']['attributes'] = elem.pop('attributes')
if 'functions' in elem: elem['type']['functions'] = elem.pop('functions')
if 'c_is_class' in elem: elem['type']['c_is_class'] = elem.pop('c_is_class')
if 'values' in elem: elem['type']['values'] = elem.pop('values')
if 'flags' in elem: elem['type']['flags'] = elem.pop('flags')
if 'nullflag' in elem: elem['type']['nullflag'] = elem.pop('nullflag')
elem['name'] = name
elem['fullname'] = join_name(parent['fullname'], name)
elem['parent'] = parent
elem['typeargs'] = elem.get('typeargs', {})
elem['c_name'] = elem.get('c_name', None) or (elem['name'] + ('_' if c_is_class else ''))
if ('c_getter' in elem) or ('c_setter' in elem):
elem['c_getter'] = elem.get('c_getter', elem['c_name'])
elem['c_setter'] = elem.get('c_setter', elem['c_name'] + ' = ')
if isinstance(elem['type'], str) and elem['type'].startswith('readonly '):
elem['typeargs']['fibre.Property.mode'] = 'readonly'
elem['typeargs']['fibre.Property.type'] = elem['type'][len('readonly '):]
elem['type'] = 'fibre.Property'
if elem['typeargs']['fibre.Property.mode'] == 'readonly' and 'c_setter' in elem: elem.pop('c_setter')
elif ('flags' in elem['type']) or ('values' in elem['type']):
elem['typeargs']['fibre.Property.mode'] = elem['typeargs'].get('fibre.Property.mode', None) or 'readwrite'
elem['typeargs']['fibre.Property.type'] = regularize_valuetype(parent['fullname'], to_pascal_case(name), elem['type'])
elem['type'] = 'fibre.Property'
if elem['typeargs']['fibre.Property.mode'] == 'readonly' and 'c_setter' in elem: elem.pop('c_setter')
else:
elem['type'] = regularize_interface(parent['fullname'], to_pascal_case(name), elem['type'])
return elem
def regularize_interface(path, name, elem):
if elem is None:
elem = {}
if isinstance(elem, str):
return elem # will be resolved during type resolution
#if path is None:
# max_anonymous_type = max([int((re.findall('^' + join_name(path, 'AnonymousType') + '([1-9]+)$', x) + ['0'])[0]) for x in interfaces.keys()])
# path = 'AnonymousType' + str(max_anonymous_type + 1)
elem['name'] = split_name(name)[-1]
elem['fullname'] = path = join_name(path, name)
elem['c_name'] = elem.get('c_name', elem['fullname'].replace('.', 'Intf::')) + 'Intf'
interfaces[path] = elem
elem['functions'] = OrderedDict((name, regularize_func(path, name, func, {'obj': {'type': make_ref_type(elem)}}))
for name, func in get_dict(elem, 'functions').items())
if not 'c_is_class' in elem:
raise Exception(elem)
treat_as_class = elem['c_is_class'] # TODO: add command line arg to make this selectively optional
elem['attributes'] = OrderedDict((name, regularize_attribute(elem, name, prop, treat_as_class))
for name, prop in get_dict(elem, 'attributes').items())
elem['interfaces'] = []
elem['enums'] = []
return elem
def regularize_valuetype(path, name, elem):
if elem is None:
elem = {}
if isinstance(elem, str):
return elem # will be resolved during type resolution
elem['name'] = split_name(name)[-1]
elem['fullname'] = path = join_name(path, name)
elem['c_name'] = elem.get('c_name', elem['fullname'].replace('.', 'Intf::'))
value_types[path] = elem
if 'flags' in elem: # treat as flags
bit = 0
for k, v in elem['flags'].items():
elem['flags'][k] = elem['flags'][k] or OrderedDict()
elem['flags'][k]['name'] = k
current_bit = elem['flags'][k].get('bit', bit)
elem['flags'][k]['bit'] = current_bit
elem['flags'][k]['value'] = 0 if current_bit is None else (1 << current_bit)
bit = bit if current_bit is None else current_bit + 1
if 'nullflag' in elem:
elem['flags'] = OrderedDict([(elem['nullflag'], {'value': 0, 'bit': None}), *elem['flags'].items()])
elem['values'] = elem['flags']
elem['is_flags'] = True
elem['is_enum'] = True
enums[path] = elem
elif 'values' in elem: # treat as enum
val = 0
for k, v in elem['values'].items():
elem['values'][k] = elem['values'][k] or OrderedDict()
elem['values'][k]['name'] = k
val = elem['values'][k].get('value', val)
elem['values'][k]['value'] = val
val += 1
enums[path] = elem
elem['is_enum'] = True
return elem
def resolve_interface(scope, name, typeargs):
"""
Resolves a type name (i.e. interface name or value type name) given as a
string to an interface object. The innermost scope is searched first.
At every scope level, if no matching interface is found, it is checked if a
matching value type exists. If so, the interface type fibre.Property<value_type>
is returned.
"""
if not isinstance(name, str):
return name
if 'fibre.Property.type' in typeargs:
typeargs['fibre.Property.type'] = resolve_valuetype(scope, typeargs['fibre.Property.type'])
scope = scope.split('.')
for probe_scope in [join_name(*scope[:(len(scope)-i)]) for i in range(len(scope)+1)]:
probe_name = join_name(probe_scope, name)
#print('probing ' + probe_name)
if probe_name in interfaces:
return interfaces[probe_name]
elif probe_name in value_types:
typeargs['fibre.Property.type'] = value_types[probe_name]
return make_property_type(typeargs)
elif probe_name in generics:
return generics[probe_name](typeargs)
raise Exception('could not resolve type {} in {}. Known interfaces are: {}. Known value types are: {}'.format(name, join_name(*scope), list(interfaces.keys()), list(value_types.keys())))
def resolve_valuetype(scope, name):
"""
Resolves a type name given as a string to the type object.
The innermost scope is searched first.
"""
if not isinstance(name, str):
return name
scope = scope.split('.')
for probe_scope in [join_name(*scope[:(len(scope)-i)]) for i in range(len(scope)+1)]:
probe_name = join_name(probe_scope, name)
if probe_name in value_types:
return value_types[probe_name]
raise Exception('could not resolve type {} in {}. Known value types are: {}'.format(name, join_name(*scope), list(value_types.keys())))
def map_to_fibre01_type(t):
if t.get('is_enum', False):
return 'int32'
elif t['fullname'] == 'float32':
return 'float'
return t['fullname']
def generate_endpoint_for_property(prop, attr_bindto, idx):
prop_intf = interfaces[prop['type']['fullname']]
endpoint = {
'id': idx,
'function': prop_intf['functions']['read' if prop['type']['mode'] == 'readonly' else 'exchange'],
'in_bindings': OrderedDict([('obj', attr_bindto)]),
'out_bindings': OrderedDict()
}
endpoint_definition = {
'name': prop['name'],
'id': idx,
'type': map_to_fibre01_type(prop['type']['value_type']),
'access': 'r' if prop['type']['mode'] == 'readonly' else 'rw',
}
return endpoint, endpoint_definition
def generate_endpoint_table(intf, bindto, idx):
"""
Generates a Fibre v0.1 endpoint table for a given interface.
This will probably be deprecated in the future.
The object must have no circular property types (i.e. A.b has type B and B.a has type A).
"""
endpoints = []
endpoint_definitions = []
cnt = 0
for k, prop in intf['attributes'].items():
property_value_type = re.findall('^fibre\.Property<([^>]*), (readonly|readwrite)>$', prop['type']['fullname'])
#attr_bindto = join_name(bindto, bindings_map.get(join_name(intf['fullname'], k), k + ('_' if len(intf['functions']) or (intf['fullname'] in treat_as_classes) else '')))
attr_bindto = intf['c_name'] + '::get_' + prop['name'] + '(' + bindto + ')'
if len(property_value_type):
# Special handling for Property<...> attributes: they resolve to one single endpoint
endpoint, endpoint_definition = generate_endpoint_for_property(prop, attr_bindto, idx + cnt)
endpoints.append(endpoint)
endpoint_definitions.append(endpoint_definition)
cnt += 1
else:
inner_endpoints, inner_endpoint_definitions, inner_cnt = generate_endpoint_table(prop['type'], attr_bindto, idx + cnt)
endpoints += inner_endpoints
endpoint_definitions.append({
'name': k,
'type': 'object',
'members': inner_endpoint_definitions
})
cnt += inner_cnt
for k, func in intf['functions'].items():
endpoints.append({
'id': idx + cnt,
'function': func,
'in_bindings': OrderedDict([('obj', bindto), *[(k_arg, '(' + bindto + ')->' + func['name'] + '_in_' + k_arg + '_') for k_arg in list(func['in'].keys())[1:]]]),
'out_bindings': OrderedDict((k_arg, '&(' + bindto + ')->' + func['name'] + '_out_' + k_arg + '_') for k_arg in func['out'].keys()),
})
in_def = []
out_def = []
for i, (k_arg, arg) in enumerate(list(func['in'].items())[1:]):
endpoint, endpoint_definition = generate_endpoint_for_property({
'name': arg['name'],
'type': make_property_type({'fibre.Property.type': arg['type'], 'fibre.Property.mode': 'readwrite'})
}, intf['c_name'] + '::get_' + func['name'] + '_in_' + k_arg + '_' + '(' + bindto + ')', idx + cnt + 1 + i)
endpoints.append(endpoint)
in_def.append(endpoint_definition)
for i, (k_arg, arg) in enumerate(func['out'].items()):
endpoint, endpoint_definition = generate_endpoint_for_property({
'name': arg['name'],
'type': make_property_type({'fibre.Property.type': arg['type'], 'fibre.Property.mode': 'readonly'})
}, intf['c_name'] + '::get_' + func['name'] + '_out_' + k_arg + '_' + '(' + bindto + ')', idx + cnt + len(func['in']) + i)
endpoints.append(endpoint)
out_def.append(endpoint_definition)
endpoint_definitions.append({
'name': k,
'id': idx + cnt,
'type': 'function',
'inputs': in_def,
'outputs': out_def
})
cnt += len(func['in']) + len(func['out'])
return endpoints, endpoint_definitions, cnt
# Parse arguments
parser = argparse.ArgumentParser(description="Gernerate code from YAML interface definitions")
parser.add_argument("--version", action="store_true",
help="print version information")
parser.add_argument("-v", "--verbose", action="store_true",
help="print debug information (on stderr)")
parser.add_argument("-d", "--definitions", type=argparse.FileType('r', encoding='utf-8'), nargs='+',
help="the YAML interface definition file(s) used to generate the code")
parser.add_argument("-t", "--template", type=argparse.FileType('r', encoding='utf-8'),
help="the code template")
group = parser.add_mutually_exclusive_group(required=True)
group.add_argument("-o", "--output", type=argparse.FileType('w', encoding='utf-8'),
help="path of the generated output")
group.add_argument("--outputs", type=str,
help="path pattern for the generated outputs. One output is generated for each interface. Use # as placeholder for the interface name.")
parser.add_argument("--generate-endpoints", type=str, nargs='?',
help="if specified, an endpoint table will be generated and passed to the template for the specified interface")
args = parser.parse_args()
if args.version:
print("0.0.1")
sys.exit(0)
definition_files = args.definitions
template_file = args.template
# Load definition files
for definition_file in definition_files:
try:
file_content = yaml.load(definition_file, Loader=SafeLineLoader)
except yaml.scanner.ScannerError as ex:
print("YAML parsing error: " + str(ex), file=sys.stderr)
sys.exit(1)
for err in validator.iter_errors(file_content):
if '__line__' in err.absolute_path:
continue
if '__column__' in err.absolute_path:
continue
#instance = err.instance.get(re.findall("([^']*)' (?:was|were) unexpected\)", err.message)[0], err.instance)
# TODO: print line number
raise Exception(err.message + '\nat ' + str(list(err.absolute_path)))
interfaces.update(get_dict(file_content, 'interfaces'))
value_types.update(get_dict(file_content, 'valuetypes'))
dictionary += file_content.get('dictionary', None) or []
# Preprocess definitions
# Regularize everything into a wellknown form
for k, item in list(interfaces.items()):
regularize_interface('', k, item)
for k, item in list(value_types.items()):
regularize_valuetype('', k, item)
if args.verbose:
print('Known interfaces: ' + ''.join([('\n ' + k) for k in interfaces.keys()]))
print('Known value types: ' + ''.join([('\n ' + k) for k in value_types.keys()]))
clashing_names = list(set(value_types.keys()).intersection(set(interfaces.keys())))
if len(clashing_names):
print("**Error**: Found both an interface and a value type with the name {}. This is not allowed, interfaces and value types (such as enums) share the same namespace.".format(clashing_names[0]), file=sys.stderr)
sys.exit(1)
# Resolve all types into references
for _, item in list(interfaces.items()):
for _, prop in item['attributes'].items():
prop['type'] = resolve_interface(item['fullname'], prop['type'], prop['typeargs'])
for _, func in item['functions'].items():
for _, arg in func['in'].items():
arg['type'] = resolve_valuetype(item['fullname'], arg['type'])
for _, arg in func['out'].items():
arg['type'] = resolve_valuetype(item['fullname'], arg['type'])
# Attach interfaces to their parents
toplevel_interfaces = []
for k, item in list(interfaces.items()):
k = split_name(k)
if len(k) == 1:
toplevel_interfaces.append(item)
else:
if k[:-1] != ['fibre']: # TODO: remove special handling
parent = interfaces[join_name(*k[:-1])]
parent['interfaces'].append(item)
item['parent'] = parent
toplevel_enums = []
for k, item in list(enums.items()):
k = split_name(k)
if len(k) == 1:
toplevel_enums.append(item)
else:
if k[:-1] != ['fibre']: # TODO: remove special handling
parent = interfaces[join_name(*k[:-1])]
parent['enums'].append(item)
item['parent'] = parent
if args.generate_endpoints:
endpoints, embedded_endpoint_definitions, _ = generate_endpoint_table(interfaces[args.generate_endpoints], '&ep_root', 1) # TODO: make user-configurable
embedded_endpoint_definitions = [{'name': '', 'id': 0, 'type': 'json', 'access': 'r'}] + embedded_endpoint_definitions
endpoints = [{'id': 0, 'function': {'fullname': 'endpoint0_handler', 'in': {}, 'out': {}}, 'bindings': {}}] + endpoints
else:
embedded_endpoint_definitions = None
endpoints = None
# Render template
env = jinja2.Environment(
comment_start_string='[#', comment_end_string='#]',
block_start_string='[%', block_end_string='%]',
variable_start_string='[[', variable_end_string=']]'
)
def tokenize(text, interface, interface_transform, value_type_transform, attribute_transform):
"""
Looks for referencable tokens (interface names, value type names or
attribute names) in a documentation text and runs them through the provided
processing functions.
Tokens are detected by enclosing back-ticks (`).
interface: The interface type object that defines the scope in which the
tokens should be detected.
interface_transform: A function that takes an interface object as an argument
and returns a string.
value_type_transform: A function that takes a value type object as an argument
and returns a string.
attribute_transform: A function that takes the token strin and an attribute
object as arguments and returns a string.
"""
if text is None or isinstance(text, jinja2.runtime.Undefined):
return text
def token_transform(token):
token = token.groups()[0]
token_list = split_name(token)
# Check if this is an attribute reference
scope = interface
attr = None
while attr is None and not scope is None:
attr_intf = scope
for name in token_list:
if not name in attr_intf['attributes']:
attr = None
break
attr = attr_intf['attributes'][name]
attr_intf = attr['type']
scope = scope.get('parent', None)
if not attr is None:
return attribute_transform(token, attr)
print('Warning: cannot resolve "{}" in {}'.format(token, interface['fullname']))
return "`" + token + "`"
return re.sub(r'`([A-Za-z\._]+)`', token_transform, text)
env.filters['to_pascal_case'] = to_pascal_case
env.filters['to_camel_case'] = to_camel_case
env.filters['to_macro_case'] = to_macro_case
env.filters['to_snake_case'] = to_snake_case
env.filters['to_kebab_case'] = to_kebab_case
env.filters['first'] = lambda x: next(iter(x))
env.filters['skip_first'] = lambda x: list(x)[1:]
env.filters['to_c_string'] = lambda x: '\n'.join(('"' + line.replace('"', '\\"') + '"') for line in json.dumps(x, separators=(',', ':')).replace('{"name"', '\n{"name"').split('\n'))
env.filters['tokenize'] = tokenize
env.filters['diagonalize'] = lambda lst: [lst[:i + 1] for i in range(len(lst))]
template = env.from_string(template_file.read())
template_args = {
'interfaces': interfaces,
'value_types': value_types,
'toplevel_interfaces': toplevel_interfaces,
'endpoints': endpoints,
'embedded_endpoint_definitions': embedded_endpoint_definitions
}
if not args.output is None:
output = template.render(**template_args)
args.output.write(output)
else:
assert('#' in args.outputs)
for k, intf in interfaces.items():
if split_name(k)[0] == 'fibre':
continue # TODO: remove special case
output = template.render(interface = intf, **template_args)
with open(args.outputs.replace('#', k.lower()), 'w', encoding='utf-8') as output_file:
output_file.write(output)
for k, enum in value_types.items():
if enum.get('builtin', False) or not enum.get('is_enum', False):
continue
output = template.render(enum = enum, **template_args)
with open(args.outputs.replace('#', k.lower()), 'w', encoding='utf-8') as output_file:
output_file.write(output)
@@ -0,0 +1,211 @@
function GCCToolchain(prefix, builddir, compiler_flags, linker_flags)
-- add some default compiler flags
compiler_flags += '-fstack-usage'
local gcc_generic_compiler = function(compiler, compiler_flags, gen_su_file, src, flags, includes, outputs)
-- resolve source path
src = tostring(src)
-- convert include list to flags
inc_flags = {}
for _,inc in pairs(includes) do
inc_flags += "-I"..tostring(inc)
end
obj_file = builddir.."/"..src:gsub("/","_")..".o"
outputs.object_files += obj_file
if gen_su_file then
su_file = builddir.."/"..src:gsub("/","_")..".su"
extra_outputs = { su_file }
outputs.su_files += su_file
else
extra_outputs = {}
end
tup.frule{
inputs= { tup.getcwd()..'/'..src },
command=compiler..' -c %f '..
tostring(compiler_flags)..' '.. -- CFLAGS for this compiler
tostring(inc_flags)..' '.. -- CFLAGS for this translation unit
tostring(flags).. -- CFLAGS for this translation unit
' -o %o',
outputs={obj_file,extra_outputs=extra_outputs}
}
end
return {
compile_c = function(src, flags, includes, outputs) gcc_generic_compiler(prefix..'gcc -std=c99', compiler_flags, true, src, flags, includes, outputs) end,
compile_cpp = function(src, flags, includes, outputs) gcc_generic_compiler(prefix..'g++ -std=c++11', compiler_flags, true, src, flags, includes, outputs) end,
compile_asm = function(src, flags, includes, outputs) gcc_generic_compiler(prefix..'gcc -x assembler-with-cpp', compiler_flags, false, src, flags, includes, outputs) end,
link = function(objects, libs, output_name)
-- convert lib list to flags
lib_flags = {}
for _,inc in pairs(libs) do
lib_flags += "-l"..tostring(inc)
end
output_name = builddir..'/'..output_name
tup.frule{
inputs=objects,
command=prefix..'g++ %f '..
tostring(linker_flags)..' '..
tostring(lib_flags)..' '..
'-Wl,-Map=%O.map'..
' -o %o',
outputs={output_name..'.elf', extra_outputs={output_name..'.map'}}
}
-- display the size
tup.frule{inputs={output_name..'.elf'}, command=prefix..'size %f'}
-- create *.hex and *.bin output formats
tup.frule{inputs={output_name..'.elf'}, command=prefix..'objcopy -O ihex %f %o', outputs={output_name..'.hex'}}
tup.frule{inputs={output_name..'.elf'}, command=prefix..'objcopy -O binary -S %f %o', outputs={output_name..'.bin'}}
end
}
end
function LLVMToolchain(arch, compiler_flags, linker_flags)
-- add some default compiler flags
--compiler_flags += '-march='..arch
compiler_flags += '-std=c++14'
clang_generic_compiler = function(compiler, compiler_flags, src, flags, includes, outputs)
-- add includes to CFLAGS
for _,inc in pairs(includes) do
flags += "-I"..inc
end
-- todo: vary build directory
obj_file="build/"..src:gsub("/","_")..".o"
tup.frule{
inputs=src,
command=compiler..' -c %f '..
tostring(compiler_flags)..' '.. -- CFLAGS for this compiler
tostring(flags).. -- CFLAGS for this translation unit
' -o %o',
outputs={obj_file}
}
outputs.object_files += obj_file
end
return {
compile_c = function(src, flags, includes, outputs) clang_generic_compiler('clang', compiler_flags, src, flags, includes, outputs) end,
compile_cpp = function(src, flags, includes, outputs) clang_generic_compiler('clang++', compiler_flags, src, flags, includes, outputs) end,
link = function(objects, output_name)
tup.frule{
inputs=objects,
command='clang++ %f '..
tostring(linker_flags)..
' -o %o',
outputs=output_name
}
end
}
end
function get_generalized_paths(paths)
if paths == nil then
return {}
else
-- TODO: check for string
generalized_paths = {}
for _,path in pairs(paths) do
table.insert(generalized_paths, tup.nodevariable(path))
end
return generalized_paths
end
end
-- A package is a collection of source files and associated
-- information required to compile those source files.
-- pkg.sources: The source files that shall be compiled as
-- part of this package
-- pkg.private_headers: The include directories that are required
-- to compile the source files in this package
-- pkg.headers: The include directories that are _exported_
-- by this package. These directories are included
-- when compiling other packages that import
-- this package.
-- pkg.packages: The packages that are needed to compile and link this
-- package. The public include directories of each imported
-- package are passed to the compiler when compiling
-- the source files of this package. The object files
-- emitted by the imported packages are included when linking
-- this package.
-- pkg.libs: The libraries that are needed to link this
-- package
function define_package(pkg)
--print('defined package in '..tup.getcwd())
pkg.sources = get_generalized_paths(pkg.sources)
pkg.objects = get_generalized_paths(pkg.objects)
pkg.headers = get_generalized_paths(pkg.headers)
pkg.private_headers = get_generalized_paths(private_headers)
if pkg.packages == nil then pkg.packages = {} end
if pkg.libs == nil then pkg.libs = {} end
if pkg.c_flags == nil then pkg.c_flags = {} end
if pkg.cpp_flags == nil then pkg.cpp_flags = {} end
if pkg.asm_flags == nil then pkg.asm_flags = {} end
return pkg
end
-- Builds object files from the source files in the specified package
function build_objects(pkg, toolchain)
all_headers = {}
tup.append_table(all_headers, pkg.private_headers)
tup.append_table(all_headers, pkg.headers)
-- add exported header directories of each imported package
for _,imported_pkg in pairs(pkg.packages) do
tup.append_table(all_headers, imported_pkg.headers)
end
-- compile
outputs = {
object_files = {}
}
tup.append_table(outputs.object_files, pkg.objects)
for _,src in pairs(pkg.sources) do
--print("compile "..src)
ext = tup.ext(tostring(src))
if ext == 'c' then
toolchain.compile_c(src, pkg.c_flags, all_headers, outputs)
elseif ext == 'cpp' then
toolchain.compile_cpp(src, pkg.cpp_flags, all_headers, outputs)
elseif ext == 's' or tup.ext(src) == 'asm' then
toolchain.compile_asm(src, pkg.asm_flags, all_headers, outputs)
else
error('unrecognized file ending')
end
end
return outputs.object_files
end
function build_executable(name, pkg, toolchain)
all_object_files = {}
all_libs = {}
-- TODO: flatten the import hierarchy prior to compiling
-- build current package
tup.append_table(all_object_files, build_objects(pkg, toolchain))
tup.append_table(all_libs, pkg.libs)
-- build imported packages
for _,imported_pkg in pairs(pkg.packages) do
objects = build_objects(imported_pkg, toolchain)
tup.append_table(all_object_files, objects)
tup.append_table(all_libs, imported_pkg.libs)
end
-- link
--tup.append_table(args.linker_objects, outputs.object_files)
print('link objects ')
print(all_object_files)
print(all_libs)
toolchain.link(all_object_files, all_libs, name)
end