Elite: Gefährlich und CosmosDB

o7 cmdr!

An einem warmen Quarantäneabend brach in einem der Telegramm-Chats zu Elite: Dangerous eine Diskussion über das Thema aus: Welche Arten von Sternen haben am häufigsten erdähnliche Planeten?

Tatsache ist, dass die Erforschung von Planeten eine der Hauptmechanismen des Spiels ist. Und in der Hierarchie der Nützlichkeit der Planeten stehen die erdähnlichen ganz oben. Ihre Seltenheit ist aber auch recht hoch. Die Kommandeure wollten also wissen: Auf welche Sterne sollten sie achten, wenn sie sich entlang der Milchstraße bewegen?

Aus dieser Diskussion entstand ein ganzes Projekt, das ich schließlich begrub. Nein, wir haben die Antwort auf die damit gestellte Frage gefunden. Aber ich mochte das Projekt aus verschiedenen Gründen nicht und nach ein paar Monaten des Aufschubs startete ich die zweite Iteration. Was daraus wurde, sowie die Antwort auf die gestellte Frage - in diesem Artikel.

Einführung

In der Elite geschieht ein großer Teil des Gameplays, was auch immer Sie denken, in einem Browser mit einem Dutzend offener Tabs. Wir werden nicht richtig argumentieren oder nicht, aber definitiv hat dies seinen eigenen Charme und viele Leute mögen es.

, , . , Frontier Developments . , .

? , Log- . .

() ( Windows):

C:\Users\User Name\Saved Games\Frontier Developments\Elite Dangerous\

. . .. PC ( ) — .

, , , . ? - , ( , , , , POI, etc).

PC:

. - . , . , , . , 400 (, ! ). , -.

, . ( , ). : , , (, ) .. , , , , — EDSM EDDB.

, , — . .

EDSM , . , , . , , — JSON 10. - - . , , - , . , ! .

% . 6 . .

, , . , , !

, , , - ( , ). EDSM, EDDB, etc. — EDDN. , , , .

, EDDN ? ...

EDDN

EDDN — :

The Elite: Dangerous Data Network is a system for willing Commanders to share dynamic data about the galaxy with others.

By pooling data in a common format, tools and analyses can be produced that add an even greater depth and vibrancy to the in-game universe.



EDDN is not run by or affiliated with Frontier Developments.

( ).

HTTP Endpoint https://eddn.edcd.io:4430/upload/ ( ).

. ZeroMQ tcp://eddn.edcd.io:9500. .

( ):

, . ? , . , ( ) ! , - .

, :

— Microsoft Azure. - . — Azure, dotnet core/standard

--- , .

, :

1. Message Distributor

EDDN Channel . ( schemaRef , ) — . EDDN 5:

• https://eddn.edcd.io/schemas/journal/1
• https://eddn.edcd.io/schemas/blackmarket/1
• https://eddn.edcd.io/schemas/commodity/3
• https://eddn.edcd.io/schemas/outfitting/2
• https://eddn.edcd.io/schemas/shipyard/2

. ? , ( ), , . journal, .. ( ). , , .

, MessageDistributor EDDN . .

2. Azure Storage Queue

( Message Processor , ). , . Storage Account ( ) connection string ( , Azure AD, ). Storage Account ( — , , ).

Azure Storage Emulator Azure Storage Explorer

3. Message Processor

, , . Azure Function App.

, - ? .

, Azure Function WebJob ( , ), . , Azure Function Runtime ( , ) — — , — . , .. — .

, , (Scale Out), , , - , CPU/Memory consumption, etc.

, (in/out bindings). , , QueueTrigger — (, dotnet, ). CosmosDBTrigger . ( ). , , CosmosDB ( ) . - , , : db client ( in-built DI, ). , , queue .

QueueTrigger. , — (invisible) . , 30 ( ). 30 — DequeueCount 1. ( -> visible state on, ++DequeueCount). DequeueCount = 5 ( ), -poison. journal, 5 , journal-poison. ( - ). , . .

: WebJob AppService. — Function App. : , .

. : , , 2 . 2 , , . , (upsert — update || insert) , , ;-) 2 , , . , . , . , , , . , - (, , CosmosDB ACID compliant).

IMHO: , . . 1. , — . (, ) .

Cosmos DB Optimistic Concurrency, . . .

4. CosmosDB

. , ( ) , . , .. .

4: Signals, Systems, Stations, Bodies. , journal ( , ? -).

, journal Event, : CarrierJump, Docked, FsdJump, Location, SaaSignalsFound, Scan ( ). , — . : — . .

CosmosDB. ( , ) Request Unit per second — RU/s. .. RU/s . 2 : Provisioned throughput Serverless ( , ). , Provisioned RU/s ( ), Serverless, RU/s , , .



Provisioned RU/s. , RU/s 400, 10% 1000% (40-4000 RU/s)



RU/s Provisioned mode — 400. , , , 250 RU/s 150 , Serverless .



, ( CosmosDB). : CosmicClone. Serverless . 2 . Provisioned Mode , , .



, zero downtime "" — , MessageDistributor , . , . . .



CosmosDB

, , : EDDN , .

, React Native — dotnet, . .

: EDDNConsumer. , , .

:

• EDDNConsumerCore — MessageDistributor. EDDN
• EDDNModels — EDDN
• JournalContributor — MessageProcessor journal
• SharedLibrary — . .

EDDNConsumerCore

dotnet core 3.1 . Main DI. HostedService ConsumerService

services.AddHostedService<ConsumerService>();

nuget- ( ): NetMQ — ZeroMq Ionic.Zlib — NetMQ

StartAsync, — NetMQRuntime ClientAsync ( , ):

private async Task ClientAsync()
{
    var utf8 = new UTF8Encoding();
    using (var client = new SubscriberSocket())
    {
        client.Connect(_eddnClientSettings.ConnectionString);
        client.SubscribeToAnyTopic();
        while (true)
        {
            try
            {
                (var bytes, _) = await client.ReceiveFrameBytesAsync();
                var uncompressed = ZlibStream.UncompressBuffer(bytes);
                var result = utf8.GetString(uncompressed);
                await _messageDistributor.DistributeAsync(result);
                _logger.LogInformation(result);
            }
            catch (Exception ex)
            {
                _logger.LogError(ex, "Error reading message queue");
            }
        }
    }
}

, ( , 1 ), _messageDistributor , , . .

MessageDistributor DistributeAsync:

public async Task DistributeAsync(string message)
{
    try
    {
        using var stringReader = new StringReader(message);
        using var jsonReader = new JsonTextReader(stringReader);
        var result = _serializer.Deserialize<Entity<BaseMessage>>(jsonReader);
        var queue = await _messageQueueFactory.GetQueueAsync(result);
        await queue.SendMessageAsync(message.Base64Encode());
    }
    catch(Exception ex)
    {
        _logger.LogError(ex, "Error distributing message");
    }
}

, , MessageQueueFactory. — . : . , schemaRef header ( test_data.json ). , , .

, ( — JSON) , . \\ , — , , null default. ( ) , POCO , , , , - . .

, , json- NewtonsoftJson ( json, ). Missing Member

_serializer.MissingMemberHandling = MissingMemberHandling.Error;
_serializer.Error += _serializer_Error;

, Application Insights, Notification Alerts . , , Header. . , .. , missing property. , , .

, header , . , ( ):

public class MessageQueueFactory : IMessageQueueFactory
{
    private readonly StorageAccount _storageOptions;
    private readonly QueueMapping _queueMapping;
    private readonly IDictionary<string, QueueClient> _queues = new Dictionary<string, QueueClient>();

    public MessageQueueFactory(
        IOptions<StorageAccount> storageOptions,
        IOptions<QueueMapping> queueMapping)
    {
        _storageOptions = storageOptions.Value;
        _queueMapping = queueMapping.Value;
    }

    public async Task<QueueClient> GetQueueAsync(Entity<BaseMessage> entity)
    {
        if (_queueMapping.TryGetValue(entity.SchemaRef, out var queueName))
        {
            if (!_queues.ContainsKey(queueName))
            {
                var client = new QueueClient(_storageOptions.StorageConnectionString, queueName);
                await client.CreateIfNotExistsAsync();
                _queues.Add(queueName, client);
            }
            return _queues[queueName];
        }
        throw new ArgumentException($"Queue {entity.SchemaRef} has not configured", nameof(entity.SchemaRef));
    }
}

, appsettings json :

{
  "QueueMapping": {
    "eddn.edcd.io/schemas/journal/1": "journal"
  }
}

( ):

"QueueMapping": {
    "eddn.edcd.io/schemas/journal/1": "journal",
    "eddn.edcd.io/schemas/blackmarket/1": "blackmarket",
    "eddn.edcd.io/schemas/commodity/3": "commodity",
    "eddn.edcd.io/schemas/outfitting/2": "outfitting",
    "eddn.edcd.io/schemas/shipyard/2": "shipyard"
  }

Dictionary<string, string> schemaRef. .

1 _queues — QueueClient, . , . , . round trip , . , , . , .

, — , — . . MessageDistributor, .

. 3 . .

EDDNModels

POCO . , . , JournalMessage:

[JsonProperty("id")]
public override string Id
{
    get => Event switch
    {
        JournalEvent.FsdJump => StarSystem,
        JournalEvent.Scan => BodyName,
        JournalEvent.Docked => StationName,
        JournalEvent.Location => BodyName,
        JournalEvent.CarrierJump => StarSystem,
        JournalEvent.SaaSignalsFound => BodyName,
        _ => $"UnknownID_{Guid.NewGuid()}"
    };
}

Id, , CosmosDB. ORM. , journal ? , . Event Id. - , - , - .

. : , , , , , , , , , \ .. ( JournalMessage). , , .. POCO — .

.

JournalContributor

"" journal — Azure Function App. Function App dotnet core class library , , . FunctionName . : — Function App ( ). - — . , . . , , ( ).

, :

public class JournalContributor
    {
        private readonly IEventTypeProcessorFactory _eventTypeProcessorFactory;

        public JournalContributor(IEventTypeProcessorFactory eventTypeProcessorFactory)
        {
            _eventTypeProcessorFactory = eventTypeProcessorFactory;
        }

        [FunctionName("ContributeJournal")]
        public async Task Run(
            [QueueTrigger("journal", Connection = "AzureWebJobsStorage")]
            Entity<JournalMessage> myQueueItem,
            ILogger log)
        {
            try
            {
                var eventProcessor = _eventTypeProcessorFactory.GetProcessor(myQueueItem.Message);
                await eventProcessor.ProcessEventAsync(myQueueItem.Message);
            }
            catch(Exception ex)
            {
                log.LogError(ex, $"Error processing queue item: {JsonConvert.SerializeObject(myQueueItem)}");
                throw;
            }
        }
    }

QueueTrigger ( ). , Event … .

, , . — , dotnet . Startup Configure, FunctionStartup (assembly):

using JournalContributor.Settings;
using Microsoft.Extensions.Configuration;
using System.IO;

[assembly: FunctionsStartup(typeof(JournalContributor.Startup))]

namespace JournalContributor
{
    public class Startup : FunctionsStartup
    {
        private IConfigurationRoot _functionConfig;
        private readonly string COSMOS_CONNECTION_STRING = Environment.GetEnvironmentVariable("CosmosDBConnectionString");
        private readonly string ENVIRONMENT = Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT");

        public override void Configure(IFunctionsHostBuilder builder)
        {
            _functionConfig = new ConfigurationBuilder()
                .AddJsonFile(Path.Combine(builder.GetContext().ApplicationRootPath, "appsettings.json"), optional: true, reloadOnChange: true)
                .AddJsonFile(Path.Combine(builder.GetContext().ApplicationRootPath, $"appsettings.{ENVIRONMENT}.json"), optional: true, reloadOnChange: true)
                .Build();

            builder.Services.AddSingleton<CosmosClient>(factory => new CosmosClient(COSMOS_CONNECTION_STRING));
            builder.Services.AddSingleton<IEventTypeProcessorFactory, EventTypeProcessorFactory>();
            builder.Services.AddTransient<FsdJumpProcessor>();
            builder.Services.AddTransient<ScanProcessor>();
            builder.Services.AddTransient<DockedProcessor>();
            builder.Services.AddTransient<LocationProcessor>();
            builder.Services.AddTransient<SaaSignalsFoundProcessor>();
            builder.Services.Configure<CosmosDbSettings>(_functionConfig.GetSection("CosmosDbSettings"));
        }
    }
}

DI . , CosmosClient : (CosmosDB). ? . 2 CosmosDB — , . , .

. , , , switch :

public IEventTypeProcessor GetProcessor(JournalMessage journalMessage) => journalMessage.Event switch
        {
            JournalEvent.FsdJump => _serviceProvider.GetService<FsdJumpProcessor>(),
            JournalEvent.Scan => _serviceProvider.GetService<ScanProcessor>(),
            JournalEvent.Docked => _serviceProvider.GetService<DockedProcessor>(),
            JournalEvent.Location => _serviceProvider.GetService<LocationProcessor>(),
            JournalEvent.SaaSignalsFound => _serviceProvider.GetService<SaaSignalsFoundProcessor>(),
            JournalEvent.CarrierJump => _serviceProvider.GetService<DockedProcessor>(),
            _ => throw new ArgumentException($"Unknown Event {journalMessage.Event}", nameof(journalMessage.Event))
        };

, , . ScanProcessor, .. 2- :

public async Task ProcessEventAsync(JournalMessage message)
        {
            var existingItem = await message.CheckIfItemExists(_bodiesContainer, _options.BodiesCollection.PartitionKey);
            if (existingItem == null)
                await _bodiesContainer.UpsertItemAsync(message);
            else
            {
                //Basic scan type contains less information then others.
                //We`re skipping item upsert if remote item has scan type higher then Basic
                //We`re also skippings if remote item scan type is Detailed (as it`s a maximum info scan)
                //Will update item if both (remote and current) have scan type Basic
                //And will upsert item in case of current item have higher scan type then remote
                if ((message.ScanType == ScanType.Basic && existingItem.ScanType != ScanType.Basic) ||
                    existingItem.ScanType == ScanType.Detailed)
                    return;
                else
                    await _bodiesContainer.UpsertItemAsync(message);
            }
        }

, , ( !) , , , , . . . , . , : , Bodies Id . , .

UpSert, Insert, .. concurrency : Item with such Id already exists ( - ). , , Optimistic Concurrency . . upsert.

, ScanType. Detailed — . Basic — — . Basic — . .

. , 3 , MessageProcessor journal .

, . , Azure WebJob EDDN , Storage Account , Azure Function App Cosmos DB . Application Insights (, -, , ). , .

continuous, .. .

![image](img src="https://habrastorage.org/webt/i_/5u/zq/i_5uzqxkanblmotxacqhtwnp_wq.png)

, , schemaRef . , .

Stream-log : , , … .

. , , .

RU/s . — Provisioned Mode 400 RU/s. , . ~105RU/s.

. 145 . EDSM, .

. Check.

. . .

. . - .

45 . , MSDN , 45 . . , . . 45 — 1.5 . , — .

— . - — , - — "" . .

. ~10 . appinsjournalweprivate . Function App. — , . \. :

( ), host.json JournalContributor:

{
  "version": "2.0",
  "logging": {
    "applicationInsights": {
      "samplingExcludedTypes": "Request",
      "samplingSettings": {
        "isEnabled": true
      }
    },
    "logLevel": {
      "default": "Information",
      "Host": "Error",
      "Function": "Error",
      "Host.Aggregator": "Information"
    }
  }
}

:

, ( 21 ):

( 193 ). - / . 21 1.75 . — appinsweprivate — WebJob. 0.98 0.19 21 .

cdbweprivate — CosmosDB Provisioned Mode 400 RU/s. . cdbslweprivate — Serverless mode CosmosDB. 4.47 .

— laweprivate. Log Analytics Workspace, , , .. CosmosDB. , , , . .

, 175,8 . , .

, . -. appinsights — 3 . , . , % , . - , , 3 — ( ).

-. CosmosDB RU/s ( , ) RU/s (Pricing), :

EDDN. , , Elite: Dangerous. - " " — . , , DLC Odyssey , . .

, Function App Storage Account, , .

, . — .

. .

.

" " " ", . - , . =) , -, ...

— . , - , 145 , EDSM 57 . - - , .

, , , . . " " — .

, — . ? . , , EDSM . , JSON — . — ( ). RU/s , , . .

CosmosDB — Jupyter Notebooks. C#, Python, - . , .

, , . ( , Sol [0,0,0]:

? :

5 :

EDSM, - , . , , — - . , , .

CosmosDB, Azure Function Change Feed ( ), .

— CosmosDB .

Für diejenigen, die bis zum Ende durchgehalten haben und immer noch daran interessiert sind, wo sie nach erdähnlichen Planeten in der Elite suchen können, ist hier Ihre Antwort (die Ergebnisse basieren auf Daten, die vor etwa sechs Monaten aus einem vollständigen EDSM-Dump stammen):

F-, K-, G-, A-Typen und aus irgendeinem Grund Neutronen bilden die fünf häufigsten Arten von Sternen, aus denen Piloten erdähnliche Planeten gefunden haben. Nun, das sind die Ergebnisse ¯ \ _ (ツ) / ¯

Gut. Das ist alles.

Flieg sicher, cmdr!