Instrumentation does not work with dotnet CLI

[RassK]

[EDIT @pjanotti: the issue is fixed for versions after [v0.6.0-beta.1(]https://github.com/open-telemetry/opentelemetry-dotnet-instrumentation/releases/tag/v0.6.0-beta.1)]

Exception:

PS C:\Users\Administrator\Desktop\OTelConsole-NET6.0> dotnet run My.Simple.Console
Unhandled exception. System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
 ---> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
 ---> System.TypeInitializationException: The type initializer for 'OpenTelemetry.AutoInstrumentation.Loader.Startup' threw an exception.
 ---> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
 ---> System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.Extensions.Configuration.Abstractions, Version=7.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. The system cannot find the file specified.
File name: 'Microsoft.Extensions.Configuration.Abstractions, Version=7.0.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'
   at Microsoft.Extensions.DependencyInjection.ProviderBuilderServiceCollectionExtensions.<>c.<AddOpenTelemetryProviderBuilderServices>b__2_0(IServiceProvider sp)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitFactory(FactoryCallSite factoryCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite callSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.Resolve(ServiceCallSite callSite, ServiceProviderEngineScope scope)
   at Microsoft.Extensions.DependencyInjection.ServiceProvider.CreateServiceAccessor(Type serviceType)
   at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
   at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(Type serviceType, ServiceProviderEngineScope serviceProviderEngineScope)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.ServiceProviderEngineScope.GetService(Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
   at OpenTelemetry.Internal.ConfigurationExtensions.<>c__DisplayClass5_0`1.<RegisterOptionsFactory>b__0(IServiceProvider sp)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitFactory(FactoryCallSite factoryCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite callSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitRootCache(ServiceCallSite callSite, RuntimeResolverContext context)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
   at Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.Resolve(ServiceCallSite callSite, ServiceProviderEngineScope scope)
   at Microsoft.Extensions.DependencyInjection.ServiceProvider.CreateServiceAccessor(Type serviceType)
   at System.Collections.Concurrent.ConcurrentDictionary`2.GetOrAdd(TKey key, Func`2 valueFactory)
   at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(Type serviceType, ServiceProviderEngineScope serviceProviderEngineScope)
   at Microsoft.Extensions.DependencyInjection.ServiceProvider.GetService(Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
   at Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService[T](IServiceProvider provider)
   at OpenTelemetry.Trace.OtlpTraceExporterHelperExtensions.<>c__DisplayClass2_0.<AddOtlpExporter>b__1(IServiceProvider sp, TracerProviderBuilder builder)
   at OpenTelemetry.ProviderBuilderServiceCollectionCallbackHelper`3.<>c__DisplayClass0_0.<RegisterConfigureBuilderCallback>b__0(IServiceProvider sp, TState state)
   at OpenTelemetry.ProviderBuilderServiceCollectionCallbackHelper`3.ConfigureProviderBuilderStateCallbackRegistration.Configure(IServiceProvider serviceProvider, TState state)
   at OpenTelemetry.ProviderBuilderServiceCollectionCallbackHelper`3.InvokeRegisteredConfigureStateCallbacks(IServiceProvider serviceProvider, TState state)
   at OpenTelemetry.Trace.TracerProviderSdk..ctor(IServiceProvider serviceProvider, Boolean ownsServiceProvider)
   at OpenTelemetry.Trace.TracerProviderBuilderBase.Build()
   at OpenTelemetry.Trace.TracerProviderBuilderBase.InvokeBuild()
   at OpenTelemetry.Trace.TracerProviderBuilderExtensions.Build(TracerProviderBuilder tracerProviderBuilder)
   at OpenTelemetry.AutoInstrumentation.Instrumentation.Initialize() in F:\Dev\forks\rassk\opentelemetry-dotnet-instrumentation\src\OpenTelemetry.AutoInstrumentation\Instrumentation.cs:line 121
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
   --- End of inner exception stack trace ---
   at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)
   at OpenTelemetry.AutoInstrumentation.Loader.Startup.TryLoadManagedAssembly() in F:\Dev\forks\rassk\opentelemetry-dotnet-instrumentation\src\OpenTelemetry.AutoInstrumentation.Loader\Startup.cs:line 71
   at OpenTelemetry.AutoInstrumentation.Loader.Startup..cctor() in F:\Dev\forks\rassk\opentelemetry-dotnet-instrumentation\src\OpenTelemetry.AutoInstrumentation.Loader\Startup.cs:line 47
   --- End of inner exception stack trace ---
   at OpenTelemetry.AutoInstrumentation.Loader.Startup..ctor()
   at System.RuntimeType.CreateInstanceDefaultCtor(Boolean publicOnly, Boolean wrapExceptions)
   --- End of inner exception stack trace ---
   at System.RuntimeType.CreateInstanceDefaultCtor(Boolean publicOnly, Boolean wrapExceptions)
   at System.Activator.CreateInstance(Type type, Boolean nonPublic, Boolean wrapExceptions)
   at System.RuntimeType.CreateInstanceImpl(BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture)
   at System.Activator.CreateInstance(Type type, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes)
   at System.Reflection.Assembly.CreateInstance(String typeName, Boolean ignoreCase, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes)
   at System.Reflection.Assembly.CreateInstance(String typeName)
   at StartupHook.Initialize() in F:\Dev\forks\rassk\opentelemetry-dotnet-instrumentation\src\OpenTelemetry.AutoInstrumentation.StartupHook\StartupHook.cs:line 75
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
   --- End of inner exception stack trace ---
   at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at System.Reflection.MethodBase.Invoke(Object obj, Object[] parameters)
   at System.StartupHookProvider.CallStartupHook(StartupHookNameOrPath startupHook)
   at System.StartupHookProvider.ProcessStartupHooks()

Test App:

using System.Diagnostics;

namespace My.Simple.ConsoleApp;

internal class Program
{
    private static ActivitySource MySource = new ActivitySource("My.Simple.Console");

    public static void Main(string[] args)
    {
        Console.WriteLine("Hello My.Simple.ConsoleApp");

        using (var activity = MySource.StartActivity("Main"))
        {
            Console.WriteLine("Inside activity create");

            activity?.AddTag("my.custom.tag", "my.simple.consoleapp");
        }

        Console.WriteLine("All done");
    }
}

<Project Sdk="Microsoft.NET.Sdk">

	<PropertyGroup>
		<OutputType>exe</OutputType>
		<TargetFramework>net6.0</TargetFramework>
		<ImplicitUsings>enable</ImplicitUsings>
		<Nullable>enable</Nullable>
	</PropertyGroup>

	<ItemGroup>
		<PackageReference Include="Microsoft.Extensions.Configuration.Abstractions" Version="7.0.0" />
		<PackageReference Include="System.Diagnostics.DiagnosticSource" Version="7.0.0" />
	</ItemGroup>

</Project>

[RassK]

I guess it's the same thing as previously was with dotnet help, but we did not fix it.

[pellared]

From #1699 (comment)

It [dotnet.exe] is a self-contained executable. Our implementation does not work for self-contained apps as they don't support AdditionalDeps.

[RassK]

Setting this helped.
$env:OTEL_DOTNET_AUTO_EXCLUDE_PROCESSES = "dotnet,dotnet.exe"

I think we should consider baking some well known exceptions into our solution.

[RassK]

SIG: Docker templates are used to do "dotnet myapp.dll", so blocking this command will be problematic.

[pellared]

@RassK Do you think adding adding a section about dotnet run in "Troubleshooting" would help?

[RassK]

@RassK Do you think adding adding a section about dotnet run in "Troubleshooting" would help?

Needs further investigations what are the workarounds (eg compiling .exe and running directly that instead of dotnet smth)

[nrcventura]

SIG: Docker templates are used to do "dotnet myapp.dll", so blocking this command will be problematic.

I added the exclude process support to the startup hook specifically for this scenario, and adding dotnet,dotnet.exe to the exclude list allowed dotnet myapp.dll to work back then. I remember that command resulting in the startup hook executing twice. Once for dotnet.exe, and then for myapp.dll.

[pjanotti]

Skip from StartupHook is working correctly regarding the excluded processes. However, it seems that we should let the StartupHook deal with DOTNET_ADDITIONAL_DEPS and DOTNET_SHARED_STORE to avoid the error above. Otherwise, it becomes impossible to use dotnet CLI. That said per my experiments we are not out of the woods even regarding dotnet run because the application itself is still crashing due to missing dependencies.